Nov 04 00:59:39.013180 ip-10-0-140-186 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 04 00:59:39.013193 ip-10-0-140-186 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 04 00:59:39.013202 ip-10-0-140-186 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 04 00:59:39.013523 ip-10-0-140-186 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 04 00:59:49.035325 ip-10-0-140-186 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 04 00:59:49.035342 ip-10-0-140-186 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f4fc85fafc9f4990b2322e021380871e --
Nov 04 01:02:19.677284 ip-10-0-140-186 systemd[1]: Starting Kubernetes Kubelet...
Nov 04 01:02:20.010697 ip-10-0-140-186 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 04 01:02:20.010697 ip-10-0-140-186 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 04 01:02:20.010697 ip-10-0-140-186 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 04 01:02:20.010697 ip-10-0-140-186 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 04 01:02:20.016168 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016077    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.016168 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016165    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.016168 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016170    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.016168 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016173    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.016168 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016176    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016179    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016182    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016185    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016187    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016190    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016192    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016195    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016198    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016200    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016203    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016206    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016208    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016211    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016213    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016216    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016218    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016226    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016231    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.016362 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016235    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016238    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016241    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016245    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016249    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016252    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016255    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016257    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016260    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016263    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016266    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016268    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016271    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016273    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016278    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016281    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016284    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016287    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016289    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.016805 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016292    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016295    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016297    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016300    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016303    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016306    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016309    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016311    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016315    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016318    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016320    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016323    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016326    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016329    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016331    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016334    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016337    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016339    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016342    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.017267 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016345    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016347    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016350    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016353    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016355    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016358    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016361    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016364    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016367    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016371    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016374    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016377    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016380    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016383    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016385    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016388    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016390    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016393    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016396    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016398    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016401    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.017756 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016403    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016406    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016409    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016412    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016414    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016417    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016419    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016422    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016426    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016428    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016431    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016433    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016436    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016438    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016441    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016444    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016447    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016449    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016452    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.018270 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016454    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016457    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016460    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016463    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.016466    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017257    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017264    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017267    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017270    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017275    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017280    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017283    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017286    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017289    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017292    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017296    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017298    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017301    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017303    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.018718 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017306    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017309    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017311    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017314    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017316    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017319    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017322    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017325    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017328    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017330    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017333    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017335    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017338    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017341    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017344    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017347    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017350    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017353    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017355    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017358    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.019188 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017360    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017363    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017365    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017368    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017370    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017373    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017385    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017388    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017390    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017394    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017397    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017400    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017403    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017405    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017408    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017411    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017413    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017416    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017419    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.019663 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017421    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017424    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017426    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017429    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017432    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017434    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017437    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017440    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017442    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017445    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017448    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017450    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017453    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017456    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017458    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017461    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017463    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017466    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017468    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017471    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.020176 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017473    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017476    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017478    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017482    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017484    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017488    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017491    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017493    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017496    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017499    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017501    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017505    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017509    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017512    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017514    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017517    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017520    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017523    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017525    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.020646 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017528    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017531    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017534    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017536    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017539    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017541    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017544    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017546    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017549    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017551    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017554    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017557    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017559    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.017563    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018080    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018087    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018091    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018094    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018097    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.021113 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018100    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018102    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018105    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018108    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018111    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018114    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018116    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018119    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018122    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018124    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018127    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018130    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018133    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018136    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018138    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018141    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018146    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018149    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018152    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018154    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.021574 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018157    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018159    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018162    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018164    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018167    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018169    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018172    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018174    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018177    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018180    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018183    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018186    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018190    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018194    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018197    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018200    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018203    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018207    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018210    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.022055 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018213    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018216    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018219    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018221    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018224    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018226    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018230    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018232    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018235    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018239    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018242    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018245    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018247    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018250    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018252    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018255    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018257    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018260    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018263    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.022530 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018265    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018268    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018271    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018273    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018276    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018280    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018282    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018285    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018288    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018290    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018293    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018295    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018298    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018300    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018303    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018305    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018309    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018311    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018314    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018317    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.022982 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018320    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018322    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018326    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018329    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018332    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018334    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018337    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018340    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018342    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018345    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018347    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018350    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018352    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018355    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018358    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018362    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018364    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018367    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018370    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018372    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.023474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018375    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018377    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.018380    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019191    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019201    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019208    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019213    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019218    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019221    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019225    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019233    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019241    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019244    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019247    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019252    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019255    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019258    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019261    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019264    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019267    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019270    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019274    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019276    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019280    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 04 01:02:20.023949 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019283    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019286    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019290    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019293    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019296    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019299    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019302    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019313    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019316    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019319    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019322    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019327    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019330    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019333    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019336    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019339    2579 flags.go:64] FLAG: --enable-server="true"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019342    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019346    2579 flags.go:64] FLAG: --event-burst="100"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019351    2579 flags.go:64] FLAG: --event-qps="50"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019354    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019357    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019360    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019365    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019368    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019371    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 04 01:02:20.024536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019374    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019378    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019381    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019384    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019387    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019390    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019393    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019396    2579 flags.go:64] FLAG: --feature-gates=""
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019401    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019404    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019407    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019410    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019413    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019416    2579 flags.go:64] FLAG: --help="false"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019419    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019423    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019426    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019429    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019432    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019436    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019439    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019441    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019444    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 04 01:02:20.025161 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019447    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019450    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019455    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019458    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019461    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019464    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019467    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019470    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019473    2579 flags.go:64] FLAG: --lock-file=""
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019476    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019479    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019482    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019487    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019490    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019493    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019496    2579 flags.go:64] FLAG: --logging-format="text"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019499    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019503    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019506    2579 flags.go:64] FLAG: --manifest-url=""
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019508    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019513    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019517    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019521    2579 flags.go:64] FLAG: --max-pods="110"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019525    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019527    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 04 01:02:20.025746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019531    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019534    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019537    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019540    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019543    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019550    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019554    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019557    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019560    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019564    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019567    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019570    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019573    2579 flags.go:64] FLAG: --port="10250"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019576    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019579    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c3fb62457c124b40"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019582    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019585    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019588    2579 flags.go:64] FLAG: --register-node="true"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019591    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019595    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019598    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019601    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019604    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019608    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019611    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019614    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 04 01:02:20.026356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019617    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019620    2579 flags.go:64] FLAG: --runonce="false"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019622    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019626    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019629    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019631    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019634    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019638    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019641    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019644    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019647    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019650    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019655    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019658    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019661    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019666    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019669    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019674    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019677    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019680    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019684    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019687    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019690    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019693    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019696    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 04 01:02:20.026971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019699    2579 flags.go:64] FLAG: --v="2"
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019704    2579 flags.go:64] FLAG: --version="false"
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019708    2579 flags.go:64] FLAG: --vmodule=""
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019712    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.019715    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019863    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019867    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019870    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019873    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019876    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019879    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019881    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019885    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019889    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019892    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019895    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019898    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019900    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019903    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019907    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.027601 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019910    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019912    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019917    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019920    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019922    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019925    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019928    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019930    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019933    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019935    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019938    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019940    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019943    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019945    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019948    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019951    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019953    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019956    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019959    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019961    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.028149 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019964    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019966    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019969    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019971    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019974    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019976    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019979    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019981    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.019984    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020000    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020003    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020008    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020011    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020014    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020018    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020021    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020023    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020026    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020028    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.028617 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020031    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020034    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020036    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020039    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020041    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020044    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020047    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020049    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020052    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020054    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020057    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020060    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020062    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020065    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020067    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020070    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020072    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020075    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020079    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020083    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.029127 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020086    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020089    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020092    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020095    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020099    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020101    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020104    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020108    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020111    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020113    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020116    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020118    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020121    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020123    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020126    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020129    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020132    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020134    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020137    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.029616 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020139    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020142    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020145    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020147    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020150    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020152    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020155    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020157    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020160    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020163    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020165    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020168    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020170    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.020198    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020234    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020239    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.030106 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020242    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020245    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020250    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020253    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020256    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020258    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020261    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020264    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020266    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020269    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020271    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020274    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020276    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020279    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020281    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020284    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020287    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020289    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020292    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020295    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.030494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020298    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020300    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020303    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020305    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020308    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020311    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020314    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020316    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020318    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020321    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020324    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020326    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020328    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020332    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020336    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020339    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020341    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020344    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020347    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020349    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.030964 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020352    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020356    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020360    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020364    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020368    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020372    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020376    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020381    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020385    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020388    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020391    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020394    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020396    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020399    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020402    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020404    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020407    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020410    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020412    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.031456 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020415    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020417    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020420    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020423    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020425    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020428    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020432    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020436    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020439    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020442    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020445    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020447    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020450    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020452    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020455    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020458    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020460    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020463    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020465    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020468    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.031916 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020471    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020473    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020476    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020480    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020483    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020486    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020489    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020492    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020494    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020497    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020499    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020501    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020504    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020507    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020509    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020512    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020515    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020518    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020520    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.032453 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020524    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020528    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020530    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020533    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020536    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.020538    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.027523    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.027538    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.027554    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.027559    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027624    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027629    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027633    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027635    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027638    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027641    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027643    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027646    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027649    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.032910 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027652    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027654    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027657    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027659    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027662    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027665    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027667    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027670    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027672    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027675    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027677    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027680    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027682    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027686    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027694    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027698    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027700    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027703    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027705    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027708    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.033385 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027711    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027713    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027716    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027718    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027721    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027723    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027726    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027728    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027731    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027734    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027736    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027739    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027742    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027746    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027750    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027753    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027756    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027759    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027761    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.033850 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027764    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027767    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027769    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027772    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027774    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027777    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027779    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027782    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027792    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027794    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027797    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027800    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027802    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027805    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027808    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027810    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027813    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027815    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027818    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027820    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.034312 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027823    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027825    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027828    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027831    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027834    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027836    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027839    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027841    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027844    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027846    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027849    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027852    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027854    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027857    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027859    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027862    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027864    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027867    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027870    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.035068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027872    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027875    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027883    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027886    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027889    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027891    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027894    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027896    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027899    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027902    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027904    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027907    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027909    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027912    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027915    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027919    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027923    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027926    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.035632 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.027929    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.027957    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028179    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028190    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028194    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028197    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028200    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028202    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028205    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028208    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028210    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028214    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028218    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028221    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028223    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.036078 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028226    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028229    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028238    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028241    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028243    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028246    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028248    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028251    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028253    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028256    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028259    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028261    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028264    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028266    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028269    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028271    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028274    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028276    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028278    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028281    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.036426 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028286    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028289    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028291    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028294    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028296    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028298    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028301    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028303    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028306    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028308    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028311    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028313    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028316    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028319    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028327    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028329    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028332    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028334    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028337    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.036898 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028340    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028342    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028345    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028347    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028350    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028352    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028355    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028357    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028359    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028362    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028364    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028367    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028371    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028375    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028378    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028381    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028385    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028387    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028390    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.037383 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028392    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028395    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028397    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028399    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028402    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028405    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028408    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028410    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028412    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028421    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028424    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028426    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028429    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028432    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028435    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028437    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028440    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028442    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028445    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028447    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.037855 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028450    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028452    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028455    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028457    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028460    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028462    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028465    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028467    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028469    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028472    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028474    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028477    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028479    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028482    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.028505    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028553    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:02:20.038523 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028559    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028561    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028564    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028567    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028571    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028573    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028581    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028584    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028587    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028589    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028592    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028594    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028597    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028599    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028602    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028605    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028607    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028610    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028612    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028615    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:02:20.038972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028617    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028620    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028627    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028630    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028633    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028635    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028638    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028640    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028643    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028645    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028648    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028650    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028654    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028657    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028659    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028662    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028666    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028668    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028671    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:02:20.039575 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028680    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028683    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028685    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028688    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028690    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028693    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028696    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028698    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028702    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028706    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028709    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028713    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028715    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028718    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028720    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028723    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028725    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028728    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028731    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:02:20.040068 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028733    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028736    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028738    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028741    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028743    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028746    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028750    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028753    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028756    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028758    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028764    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028766    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028769    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028772    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028780    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028783    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028785    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028788    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028790    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:02:20.040521 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028793    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028796    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028798    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028801    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028803    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028805    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028808    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028810    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028813    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028815    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028818    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028820    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028823    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028826    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028828    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028831    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028833    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028836    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028838    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028842    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:02:20.040949 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028850    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028855    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028858    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028862    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028864    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028867    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028869    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:20.028872    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.028925    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.031018    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.031803    2579 server.go:1014] "Starting client certificate rotation"
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.031907    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.031955    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.033588    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 04 01:02:20.041441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.039369    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 04 01:02:20.048065 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.048046    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 04 01:02:20.051147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.051133    2579 log.go:25] "Validated CRI v1 image API"
Nov 04 01:02:20.052100 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.052071    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 04 01:02:20.056502 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.056480    2579 fs.go:135] Filesystem UUIDs: map[12ac6bd3-ecce-4867-a439-8d2d7656d4c7:/dev/nvme0n1p4 76143f64-01f5-43ed-bb53-56d92a0799fd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 04 01:02:20.056583 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.056501    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 04 01:02:20.061483 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.061361    2579 manager.go:217] Machine: {Timestamp:2026-11-04 01:02:20.060186413 +0000 UTC m=+0.279890936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105997 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210c5e45ff94d7c824ac407ce6b64c SystemUUID:ec210c5e-45ff-94d7-c824-ac407ce6b64c BootID:f4fc85fa-fc9f-4990-b232-2e021380871e Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:dd:cf:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:dd:cf:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:e0:a9:50:c4:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 04 01:02:20.061483 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.061479    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 04 01:02:20.061635 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.061587    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 04 01:02:20.064221 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.064196    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 04 01:02:20.064350 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.064223    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-186.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 04 01:02:20.064393 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.064352    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 04 01:02:20.064393 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.064361    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 04 01:02:20.064393 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.064386    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 04 01:02:20.064476 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.064399    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 04 01:02:20.065564 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.065553    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 04 01:02:20.069354 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.069341    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 04 01:02:20.069486 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.069477    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 04 01:02:20.070126 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.070116    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 04 01:02:20.070163 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.070131    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 04 01:02:20.070163 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.070143    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 04 01:02:20.070163 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.070152    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 04 01:02:20.070163 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.070160    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 04 01:02:20.070505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.070488    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 04 01:02:20.074026 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.073223    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 04 01:02:20.075974 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.075954    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 04 01:02:20.078651 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.078634    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 04 01:02:20.079342 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079328    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 04 01:02:20.079411 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079358    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 04 01:02:20.079506 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079496    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 04 01:02:20.079552 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079512    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 04 01:02:20.079552 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079519    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 04 01:02:20.079552 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079525    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 04 01:02:20.079552 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079530    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 04 01:02:20.079552 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.079536    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 04 01:02:20.080015 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080006    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 04 01:02:20.080015 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080015    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 04 01:02:20.080073 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080022    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 04 01:02:20.080073 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080028    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 04 01:02:20.080073 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080037    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 04 01:02:20.080073 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080045    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 04 01:02:20.080560 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080552    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 04 01:02:20.080560 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.080560    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 04 01:02:20.084178 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.084167    2579 server.go:1263] "Started kubelet"
Nov 04 01:02:20.084340 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.084296    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 04 01:02:20.084379 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.084320    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 04 01:02:20.084379 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.084374    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 04 01:02:20.084917 ip-10-0-140-186 systemd[1]: Started Kubernetes Kubelet.
Nov 04 01:02:20.086180 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.086156    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 04 01:02:20.087166 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.087152    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 04 01:02:20.088770 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.088746    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-186.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 04 01:02:20.089604 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.088724    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-186.us-east-2.compute.internal.18e4b1e2fe19f9ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-186.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-186.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 01:02:20.084140527 +0000 UTC m=+0.303845052,LastTimestamp:2026-11-04 01:02:20.084140527 +0000 UTC m=+0.303845052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-186.us-east-2.compute.internal,}"
Nov 04 01:02:20.092024 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.092007    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 04 01:02:20.094340 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.094318    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 04 01:02:20.094679 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.094665    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 04 01:02:20.094977 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.094959    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.095243 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095223    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 04 01:02:20.095243 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095226    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 04 01:02:20.095366 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095252    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 04 01:02:20.095423 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095372    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 04 01:02:20.095423 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095380    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 04 01:02:20.095601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095585    2579 factory.go:55] Registering systemd factory
Nov 04 01:02:20.095649 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095621    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 04 01:02:20.095875 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095862    2579 factory.go:153] Registering CRI-O factory
Nov 04 01:02:20.095875 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095875    2579 factory.go:223] Registration of the crio container factory successfully
Nov 04 01:02:20.096002 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095922    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 04 01:02:20.096002 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095941    2579 factory.go:103] Registering Raw factory
Nov 04 01:02:20.096002 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.095953    2579 manager.go:1199] Started watching for new ooms in manager
Nov 04 01:02:20.096880 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.096864    2579 manager.go:319] Starting recovery of all containers
Nov 04 01:02:20.096933 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.096897    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-186.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 04 01:02:20.097176 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.097151    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 04 01:02:20.097291 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.097250    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-186.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 04 01:02:20.104553 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.103696    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5svhl"
Nov 04 01:02:20.110166 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.110147    2579 manager.go:324] Recovery completed
Nov 04 01:02:20.114508 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.114492    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.116247 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116233    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.116334 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116259    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.116334 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116269    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.116864 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116849    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 04 01:02:20.116864 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116861    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 04 01:02:20.116864 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116854    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.117066 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.116877    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 04 01:02:20.117971 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.117954    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.118097 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.117978    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.118097 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.118007    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.118337 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.118320    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5svhl"
Nov 04 01:02:20.118735 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.118715    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 04 01:02:20.118873 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.118795    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-186.us-east-2.compute.internal.18e4b1e30003e055  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-186.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-186.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-186.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 01:02:20.116246613 +0000 UTC m=+0.335951137,LastTimestamp:2026-11-04 01:02:20.116246613 +0000 UTC m=+0.335951137,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-186.us-east-2.compute.internal,}"
Nov 04 01:02:20.119288 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.119275    2579 policy_none.go:50] "Start"
Nov 04 01:02:20.119348 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.119295    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 04 01:02:20.119348 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.119308    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 04 01:02:20.120975 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.120914    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-186.us-east-2.compute.internal.18e4b1e3000421f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-186.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-186.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-186.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 01:02:20.116263413 +0000 UTC m=+0.335967936,LastTimestamp:2026-11-04 01:02:20.116263413 +0000 UTC m=+0.335967936,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-186.us-east-2.compute.internal,}"
Nov 04 01:02:20.121362 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.121350    2579 policy_none.go:44] "Start"
Nov 04 01:02:20.147601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.147573    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 04 01:02:20.149884 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.149864    2579 manager.go:341] "Starting Device Plugin manager"
Nov 04 01:02:20.150410 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.150389    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 04 01:02:20.150410 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.150406    2579 server.go:87] "Starting device plugin registration server"
Nov 04 01:02:20.150634 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.150621    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 04 01:02:20.150692 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.150634    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 04 01:02:20.150740 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.150727    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 04 01:02:20.150819 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.150803    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 04 01:02:20.150819 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.150816    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 04 01:02:20.151314 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.151280    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 04 01:02:20.151367 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.151326    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.251343 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.251317    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.252300 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.252284    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.252380 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.252315    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.252380 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.252324    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.252380 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.252351    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.259073 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.259058    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.259121 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.259080    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-186.us-east-2.compute.internal\": node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.276872 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.276823    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.377829 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.377795    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.456193 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.456158    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 04 01:02:20.457273 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.457253    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 04 01:02:20.457273 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.457272    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 04 01:02:20.457427 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.457292    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 04 01:02:20.457908 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.457322    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 04 01:02:20.461330 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.461315    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:02:20.478480 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.478462    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.558052 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.557968    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"]
Nov 04 01:02:20.558130 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.558064    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.559014 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.559001    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.559069 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.559030    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.559069 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.559043    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.561368 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.561355    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.561521 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.561507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.561564 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.561537    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.562095 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.562078    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.562196 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.562098    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.562196 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.562112    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.562196 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.562116    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.562196 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.562126    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.562196 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.562126    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.564384 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.564368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.564474 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.564393    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:02:20.565119 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.565103    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:02:20.565211 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.565138    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:02:20.565211 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.565153    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:02:20.579094 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.579076    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.594826 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.594807    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.597382 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.597359    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.598469 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.598452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/388b6cc67d09cb3e8a1219d1e84089fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"388b6cc67d09cb3e8a1219d1e84089fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.598536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.598486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.598536 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.598516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.679744 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.679720    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.699074 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.699041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.699206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.699086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.699206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.699112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/388b6cc67d09cb3e8a1219d1e84089fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"388b6cc67d09cb3e8a1219d1e84089fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.699206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.699134    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.699206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.699147    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.699206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.699178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/388b6cc67d09cb3e8a1219d1e84089fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"388b6cc67d09cb3e8a1219d1e84089fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.780236 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.780211    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.880780 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.880708    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:20.899069 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.899031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.901209 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:20.901195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:20.980966 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:20.980777    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:21.010580 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.010550    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:02:21.032699 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.032670    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 04 01:02:21.033171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.032806    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 04 01:02:21.033171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.032844    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 04 01:02:21.082075 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:21.082046    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:02:21.107414 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.107388    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:02:21.123692 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.123661    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:57:20 +0000 UTC" deadline="2028-08-21 02:38:59.132588552 +0000 UTC"
Nov 04 01:02:21.123692 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.123686    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15745h36m38.008905135s"
Nov 04 01:02:21.195687 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.195616    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:21.208985 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.208965    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 04 01:02:21.210669 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.210656    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:02:21.218742 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.218729    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 04 01:02:21.550599 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.550460    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:02:21.633654 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:21.633621    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388b6cc67d09cb3e8a1219d1e84089fb.slice/crio-1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae WatchSource:0}: Error finding container 1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae: Status 404 returned error can't find the container with id 1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae
Nov 04 01:02:21.642298 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:21.642276    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 01:02:21.645494 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:21.645466    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe1c320288f5f6b9ae9ef95fa346d10.slice/crio-8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b WatchSource:0}: Error finding container 8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b: Status 404 returned error can't find the container with id 8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b
Nov 04 01:02:22.072910 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.072882    2579 apiserver.go:52] "Watching apiserver"
Nov 04 01:02:22.083703 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.083678    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 04 01:02:22.083987 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.083969    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-jxdjh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-fjpq6","openshift-network-diagnostics/network-check-target-58tg6","kube-system/global-pull-secret-syncer-x7c8v","openshift-dns/node-resolver-6tjgh","openshift-network-operator/iptables-alerter-x7xvv","openshift-ovn-kubernetes/ovnkube-node-5fq7t","kube-system/konnectivity-agent-6ch6l","openshift-image-registry/node-ca-dnjzj","openshift-multus/network-metrics-daemon-gfjzp","kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7","openshift-cluster-node-tuning-operator/tuned-vf66l"]
Nov 04 01:02:22.087894 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.087874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.090259 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.090234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.092053 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.092031    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.092250 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.092231    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 04 01:02:22.092437 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.092409    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhcr5\""
Nov 04 01:02:22.092497 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.092450    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.092970 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.092954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.094156 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.094047    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.094156 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.094052    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8pc2\""
Nov 04 01:02:22.094156 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.094081    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 04 01:02:22.094345 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.094171    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.094593 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.094580    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 04 01:02:22.095634 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.095372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:22.095634 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.095477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:22.097061 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097040    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 04 01:02:22.097168 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097093    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.097168 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097100    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.097168 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097164    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 04 01:02:22.097291 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097163    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq94r\""
Nov 04 01:02:22.097291 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097182    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 04 01:02:22.097794 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.097776    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.097894 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.097835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:22.100180 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.100154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.103368 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.102855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.105449 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.105424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.107761 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.107742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.107908 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.107885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-registration-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.107987 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.107924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-sys-fs\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.107987 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.107971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrfz\" (UniqueName: \"kubernetes.io/projected/5083bdb1-d4a2-4162-9b49-e3191fa61637-kube-api-access-xgrfz\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.108112 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-system-cni-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108112 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108187 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-dbus\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.108187 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.108265 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-device-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.108265 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-cnibin\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108265 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108372 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108372 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33e0d88d-b649-469b-905b-25cded0252f9-iptables-alerter-script\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.108372 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33e0d88d-b649-469b-905b-25cded0252f9-host-slash\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.108466 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jp9w\" (UniqueName: \"kubernetes.io/projected/33e0d88d-b649-469b-905b-25cded0252f9-kube-api-access-2jp9w\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.108466 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.108466 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108466 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czblg\" (UniqueName: \"kubernetes.io/projected/bd730006-869f-4fb3-b626-912e7530f33c-kube-api-access-czblg\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.108466 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-socket-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.108631 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.108631 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:22.108631 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.108606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-kubelet-config\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.110169 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.110151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.112637 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.112620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.112738 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.112704    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:22.114877 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.114859    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 04 01:02:22.115171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.115154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.135383 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.135363    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5jnlp"
Nov 04 01:02:22.148213 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.148197    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5jnlp"
Nov 04 01:02:22.196621 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.196591    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 04 01:02:22.200896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.200863    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.201017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.200969    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.201168 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.201148    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 04 01:02:22.201246 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.201187    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 04 01:02:22.201246 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.201203    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.201246 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.201221    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.202656 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.202642    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 04 01:02:22.203107 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203091    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 04 01:02:22.203171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203102    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 04 01:02:22.203171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203117    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.203171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203136    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n5n45\""
Nov 04 01:02:22.203171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203137    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.203171 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203165    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 04 01:02:22.203335 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203173    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dh9vh\""
Nov 04 01:02:22.203335 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203180    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78n7j\""
Nov 04 01:02:22.203335 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.203207    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 04 01:02:22.204280 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.204267    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 04 01:02:22.204322 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.204278    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 04 01:02:22.204356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.204334    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cfm9p\""
Nov 04 01:02:22.204407 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.204395    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pvmvv\""
Nov 04 01:02:22.204794 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.204782    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zw2l8\""
Nov 04 01:02:22.205216 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.205201    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 04 01:02:22.209200 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-node-log\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.209292 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.209292 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-k8s-cni-cncf-io\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.209292 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdm5n\" (UniqueName: \"kubernetes.io/projected/4b6e932c-bb1d-424e-8dd2-c92797151de6-kube-api-access-zdm5n\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.209292 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvg8v\" (UniqueName: \"kubernetes.io/projected/95384914-9263-45e9-9639-3566767d7559-kube-api-access-bvg8v\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.209505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33e0d88d-b649-469b-905b-25cded0252f9-iptables-alerter-script\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.209505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jp9w\" (UniqueName: \"kubernetes.io/projected/33e0d88d-b649-469b-905b-25cded0252f9-kube-api-access-2jp9w\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.209505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-konnectivity-ca\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.209505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.209505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-netns\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.209505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-kubelet\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.209815 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-sys\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.209815 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-socket-dir-parent\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.209815 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.209815 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-config\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.209815 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-dbus\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.209815 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-system-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.209873    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-dbus\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.209957    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:02:22.709935547 +0000 UTC m=+2.929640079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95384914-9263-45e9-9639-3566767d7559-hosts-file\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.209990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33e0d88d-b649-469b-905b-25cded0252f9-iptables-alerter-script\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysconfig\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-tmp\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22778bfc-2110-48e0-85dd-25e5ca26c161-host\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.210140 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-device-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-cnibin\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210212    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-device-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-multus-certs\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-cnibin\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-etc-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95384914-9263-45e9-9639-3566767d7559-tmp-dir\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.210574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4tb\" (UniqueName: \"kubernetes.io/projected/9e38076f-6754-46f2-b054-f0aa1976319e-kube-api-access-td4tb\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-kubernetes\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-conf\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-kubelet-config\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-kubelet\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-run\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-slash\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210745    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-env-overrides\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85l72\" (UniqueName: \"kubernetes.io/projected/552f596f-96f2-4d27-9b5a-1c87f8dc864d-kube-api-access-85l72\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210800    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-kubelet-config\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll26k\" (UniqueName: \"kubernetes.io/projected/22778bfc-2110-48e0-85dd-25e5ca26c161-kube-api-access-ll26k\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-etc-kubernetes\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-var-lib-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e38076f-6754-46f2-b054-f0aa1976319e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-host\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-hostroot\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-var-lib-kubelet\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210966    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.210973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22778bfc-2110-48e0-85dd-25e5ca26c161-serviceca\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-socket-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-systemd-units\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-netd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211106    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-socket-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-systemd\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-registration-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-sys-fs\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211238    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-registration-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-system-cni-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.211712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211354    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-sys-fs\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211401    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-system-cni-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-modprobe-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-agent-certs\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-bin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-conf-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-script-lib\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-tuned\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-log-socket\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33e0d88d-b649-469b-905b-25cded0252f9-host-slash\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211736    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czblg\" (UniqueName: \"kubernetes.io/projected/bd730006-869f-4fb3-b626-912e7530f33c-kube-api-access-czblg\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211773    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33e0d88d-b649-469b-905b-25cded0252f9-host-slash\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-cnibin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-netns\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.211964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-multus\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-ovn\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212100    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-bin\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-lib-modules\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrfz\" (UniqueName: \"kubernetes.io/projected/5083bdb1-d4a2-4162-9b49-e3191fa61637-kube-api-access-xgrfz\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-cni-binary-copy\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-daemon-config\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8ln\" (UniqueName: \"kubernetes.io/projected/e8fac715-d046-4c8f-af92-66684a66e0ae-kube-api-access-8q8ln\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.212856 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.212312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-systemd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.221768 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.221743    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 04 01:02:22.222518 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.222499    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:22.222606 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.222523    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:22.222606 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.222537    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:22.222716 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.222611    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:02:22.722593512 +0000 UTC m=+2.942298043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:22.225489 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.225469    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jp9w\" (UniqueName: \"kubernetes.io/projected/33e0d88d-b649-469b-905b-25cded0252f9-kube-api-access-2jp9w\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.225489 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.225482    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrfz\" (UniqueName: \"kubernetes.io/projected/5083bdb1-d4a2-4162-9b49-e3191fa61637-kube-api-access-xgrfz\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.225610 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.225483    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czblg\" (UniqueName: \"kubernetes.io/projected/bd730006-869f-4fb3-b626-912e7530f33c-kube-api-access-czblg\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.312897 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.312861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4tb\" (UniqueName: \"kubernetes.io/projected/9e38076f-6754-46f2-b054-f0aa1976319e-kube-api-access-td4tb\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.312897 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.312893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-kubernetes\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313113 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.312955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-kubernetes\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313113 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.312995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313113 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-conf\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313113 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-kubelet\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313113 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-run\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313113 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-slash\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-env-overrides\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85l72\" (UniqueName: \"kubernetes.io/projected/552f596f-96f2-4d27-9b5a-1c87f8dc864d-kube-api-access-85l72\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313145    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-kubelet\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll26k\" (UniqueName: \"kubernetes.io/projected/22778bfc-2110-48e0-85dd-25e5ca26c161-kube-api-access-ll26k\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-run\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-slash\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313217    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-conf\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-etc-kubernetes\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-var-lib-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e38076f-6754-46f2-b054-f0aa1976319e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-host\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-hostroot\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313338    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-var-lib-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313340    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-etc-kubernetes\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-var-lib-kubelet\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22778bfc-2110-48e0-85dd-25e5ca26c161-serviceca\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.313387 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313399    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-host\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313401    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-var-lib-kubelet\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-systemd-units\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313437    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-hostroot\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-netd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313451    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-systemd-units\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-systemd\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313512    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-netd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-systemd\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-modprobe-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313626    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-agent-certs\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-env-overrides\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-bin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.314281 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-conf-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-script-lib\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313724    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-modprobe-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-tuned\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313732    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-bin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-log-socket\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313769    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-conf-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313828    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-log-socket\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-cnibin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313923    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22778bfc-2110-48e0-85dd-25e5ca26c161-serviceca\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313938    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-netns\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313958    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-netns\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-multus\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.313992    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-cnibin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315117 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-ovn\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314075    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-multus\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-bin\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-lib-modules\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314132    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-ovn\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-cni-binary-copy\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-bin\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-daemon-config\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8ln\" (UniqueName: \"kubernetes.io/projected/e8fac715-d046-4c8f-af92-66684a66e0ae-kube-api-access-8q8ln\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314243    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-lib-modules\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-systemd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-node-log\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-k8s-cni-cncf-io\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdm5n\" (UniqueName: \"kubernetes.io/projected/4b6e932c-bb1d-424e-8dd2-c92797151de6-kube-api-access-zdm5n\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-systemd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg8v\" (UniqueName: \"kubernetes.io/projected/95384914-9263-45e9-9639-3566767d7559-kube-api-access-bvg8v\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-konnectivity-ca\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-k8s-cni-cncf-io\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.315977 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-netns\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-kubelet\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314464    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-node-log\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314283    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-script-lib\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314544    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-netns\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314589    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-kubelet\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-sys\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-socket-dir-parent\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314667    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-cni-binary-copy\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314683    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-daemon-config\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-config\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-system-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-socket-dir-parent\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314746    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-sys\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95384914-9263-45e9-9639-3566767d7559-hosts-file\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.314805    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysconfig\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.316821 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.314858    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:02:22.814841284 +0000 UTC m=+3.034545806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-system-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-tmp\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95384914-9263-45e9-9639-3566767d7559-hosts-file\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22778bfc-2110-48e0-85dd-25e5ca26c161-host\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.314973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-multus-certs\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-etc-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95384914-9263-45e9-9639-3566767d7559-tmp-dir\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315204    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22778bfc-2110-48e0-85dd-25e5ca26c161-host\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315236    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysconfig\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315283    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-config\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315291    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-etc-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-multus-certs\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315317    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-konnectivity-ca\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.315410    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95384914-9263-45e9-9639-3566767d7559-tmp-dir\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.316041    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e38076f-6754-46f2-b054-f0aa1976319e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.316864    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-agent-certs\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.317613 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.317441    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-tmp\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.318139 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.318122    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-tuned\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.326680 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.326612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85l72\" (UniqueName: \"kubernetes.io/projected/552f596f-96f2-4d27-9b5a-1c87f8dc864d-kube-api-access-85l72\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.327399 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.327374    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4tb\" (UniqueName: \"kubernetes.io/projected/9e38076f-6754-46f2-b054-f0aa1976319e-kube-api-access-td4tb\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.329129 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.329094    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:02:22.329243 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.329155    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdm5n\" (UniqueName: \"kubernetes.io/projected/4b6e932c-bb1d-424e-8dd2-c92797151de6-kube-api-access-zdm5n\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.330354 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.330332    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll26k\" (UniqueName: \"kubernetes.io/projected/22778bfc-2110-48e0-85dd-25e5ca26c161-kube-api-access-ll26k\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.332415 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.332383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg8v\" (UniqueName: \"kubernetes.io/projected/95384914-9263-45e9-9639-3566767d7559-kube-api-access-bvg8v\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.334147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.334125    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8ln\" (UniqueName: \"kubernetes.io/projected/e8fac715-d046-4c8f-af92-66684a66e0ae-kube-api-access-8q8ln\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.403751 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.403712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:02:22.414407 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.414382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:02:22.416648 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.416610    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e0d88d_b649_469b_905b_25cded0252f9.slice/crio-9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d WatchSource:0}: Error finding container 9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d: Status 404 returned error can't find the container with id 9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d
Nov 04 01:02:22.424884 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.424838    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5083bdb1_d4a2_4162_9b49_e3191fa61637.slice/crio-ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef WatchSource:0}: Error finding container ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef: Status 404 returned error can't find the container with id ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef
Nov 04 01:02:22.428991 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.428965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:02:22.430329 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.430308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:22.434169 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.434144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:02:22.436944 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.436922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxdjh"
Nov 04 01:02:22.439311 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.439288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:02:22.446837 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.446821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:02:22.452723 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.452683    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6eb6955_2fed_4ca2_9b74_e278bb5eb79c.slice/crio-cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec WatchSource:0}: Error finding container cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec: Status 404 returned error can't find the container with id cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec
Nov 04 01:02:22.454928 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.454897    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd730006_869f_4fb3_b626_912e7530f33c.slice/crio-bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3 WatchSource:0}: Error finding container bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3: Status 404 returned error can't find the container with id bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3
Nov 04 01:02:22.462433 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.462264    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fac715_d046_4c8f_af92_66684a66e0ae.slice/crio-912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49 WatchSource:0}: Error finding container 912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49: Status 404 returned error can't find the container with id 912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49
Nov 04 01:02:22.465139 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.465117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:22.465876 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.465816    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ch6l" event={"ID":"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c","Type":"ContainerStarted","Data":"cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec"}
Nov 04 01:02:22.466972 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.466902    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22778bfc_2110_48e0_85dd_25e5ca26c161.slice/crio-08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd WatchSource:0}: Error finding container 08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd: Status 404 returned error can't find the container with id 08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd
Nov 04 01:02:22.467932 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.467904    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6tjgh" event={"ID":"95384914-9263-45e9-9639-3566767d7559","Type":"ContainerStarted","Data":"0b1a5a1908961c7fe7f9e225e3cc8a299b0cd67aba699be13d27f5e07c9f785e"}
Nov 04 01:02:22.470458 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.470392    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerStarted","Data":"ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef"}
Nov 04 01:02:22.472411 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.472385    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x7xvv" event={"ID":"33e0d88d-b649-469b-905b-25cded0252f9","Type":"ContainerStarted","Data":"9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d"}
Nov 04 01:02:22.475741 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.475713    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"8fe1c320288f5f6b9ae9ef95fa346d10","Type":"ContainerStarted","Data":"8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b"}
Nov 04 01:02:22.477560 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.477510    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"388b6cc67d09cb3e8a1219d1e84089fb","Type":"ContainerStarted","Data":"1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae"}
Nov 04 01:02:22.487079 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.479226    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vf66l" event={"ID":"552f596f-96f2-4d27-9b5a-1c87f8dc864d","Type":"ContainerStarted","Data":"31161bd447db09aafc2d51eaa6c255f925020d1ebb9fe7ed01967f76d630cc8c"}
Nov 04 01:02:22.487079 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.481258    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerStarted","Data":"bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3"}
Nov 04 01:02:22.499497 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:02:22.499468    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e38076f_6754_46f2_b054_f0aa1976319e.slice/crio-600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973 WatchSource:0}: Error finding container 600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973: Status 404 returned error can't find the container with id 600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973
Nov 04 01:02:22.718275 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.718181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:22.718444 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.718358    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:22.718444 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.718425    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:02:23.718405561 +0000 UTC m=+3.938110090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:22.819067 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.819029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:22.819256 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.819103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:22.819256 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.819240    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:22.819386 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.819260    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:22.819386 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.819274    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:22.819386 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.819361    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:02:23.819341873 +0000 UTC m=+4.039046392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:22.819807 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.819780    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:22.819937 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:22.819843    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:02:23.819825295 +0000 UTC m=+4.039529809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:22.937865 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:22.937829    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:02:23.151029 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.150910    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:57:22 +0000 UTC" deadline="2028-07-09 06:27:01.474351809 +0000 UTC"
Nov 04 01:02:23.151029 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.150977    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14717h24m38.323379725s"
Nov 04 01:02:23.458288 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.458197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:23.458466 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.458354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:23.458861 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.458839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:23.459002 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.458941    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:23.497870 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.497826    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973"}
Nov 04 01:02:23.520202 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.520166    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dnjzj" event={"ID":"22778bfc-2110-48e0-85dd-25e5ca26c161","Type":"ContainerStarted","Data":"08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd"}
Nov 04 01:02:23.527530 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.527490    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxdjh" event={"ID":"e8fac715-d046-4c8f-af92-66684a66e0ae","Type":"ContainerStarted","Data":"912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49"}
Nov 04 01:02:23.727353 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.727245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:23.727519 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.727407    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:23.727519 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.727475    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:02:25.727455885 +0000 UTC m=+5.947160400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:23.827656 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.827609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:23.827821 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:23.827721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:23.827952 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.827932    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:23.828033 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.827960    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:23.828033 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.827974    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:23.828176 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.828037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:02:25.828017839 +0000 UTC m=+6.047722354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:23.828548 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.828525    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:23.828649 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:23.828590    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:02:25.82857289 +0000 UTC m=+6.048277406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:24.152208 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:24.152110    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:57:22 +0000 UTC" deadline="2028-06-13 08:44:53.726669774 +0000 UTC"
Nov 04 01:02:24.152208 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:24.152161    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14095h42m29.574513298s"
Nov 04 01:02:24.467005 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:24.466925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:24.467216 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:24.467188    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:25.459336 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:25.458267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:25.459336 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:25.458523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:25.459336 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.458663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:25.459336 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.459166    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:25.551041 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:25.550176    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"388b6cc67d09cb3e8a1219d1e84089fb","Type":"ContainerStarted","Data":"b2abef6f5151caa75e8437ad76069441f1239f318e4c54c0a0c9a768032813a2"}
Nov 04 01:02:25.744608 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:25.743898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:25.744608 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.744118    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:25.744608 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.744193    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:02:29.744172625 +0000 UTC m=+9.963877141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:25.845134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:25.845201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.845378    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.845402    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.845416    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.845471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:02:29.84545417 +0000 UTC m=+10.065158684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.845932    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:25.850460 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:25.845982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:02:29.845966723 +0000 UTC m=+10.065671249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:26.461149 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:26.461041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:26.461598 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:26.461217    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:26.554618 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:26.554579    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe1c320288f5f6b9ae9ef95fa346d10" containerID="c715bad5d1b418c381d382b5602488fe51f4f0f4494bee887e33f5077cc22ed6" exitCode=0
Nov 04 01:02:26.554772 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:26.554676    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"8fe1c320288f5f6b9ae9ef95fa346d10","Type":"ContainerDied","Data":"c715bad5d1b418c381d382b5602488fe51f4f0f4494bee887e33f5077cc22ed6"}
Nov 04 01:02:26.562383 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:26.562049    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="547ccb0f6ae7cb104e9a48d9d061a9eeca6b03d8fd5c6259930c27ded49b38e2" exitCode=0
Nov 04 01:02:26.562383 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:26.562163    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerDied","Data":"547ccb0f6ae7cb104e9a48d9d061a9eeca6b03d8fd5c6259930c27ded49b38e2"}
Nov 04 01:02:26.572737 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:26.572331    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal" podStartSLOduration=5.5723155030000004 podStartE2EDuration="5.572315503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:25.573871965 +0000 UTC m=+5.793576502" watchObservedRunningTime="2026-11-04 01:02:26.572315503 +0000 UTC m=+6.792020036"
Nov 04 01:02:27.458396 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:27.458320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:27.458553 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:27.458451    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:27.458868 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:27.458848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:27.459051 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:27.458972    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:27.568845 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:27.568806    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ch6l" event={"ID":"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c","Type":"ContainerStarted","Data":"e388e6ecb155bb8f242c995eceb4ebe2f48d1411583234e4295dc5a8e968d2a3"}
Nov 04 01:02:27.571892 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:27.571859    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"8fe1c320288f5f6b9ae9ef95fa346d10","Type":"ContainerStarted","Data":"03933e1b7206041c38b654b7b575a01f023c8e4babc31ed46a7fc3d6f4401e2b"}
Nov 04 01:02:27.587193 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:27.586439    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6ch6l" podStartSLOduration=3.224819017 podStartE2EDuration="7.586421513s" totalImagesPullingTime="4.361602496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:27.586008052 +0000 UTC m=+7.805712614" watchObservedRunningTime="2026-11-04 01:02:27.586421513 +0000 UTC m=+7.806126090"
Nov 04 01:02:27.606709 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:27.605852    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" podStartSLOduration=6.605815878 podStartE2EDuration="6.605815878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:27.605387544 +0000 UTC m=+7.825092079" watchObservedRunningTime="2026-11-04 01:02:27.605815878 +0000 UTC m=+7.825520413"
Nov 04 01:02:28.262239 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:28.262205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:28.458214 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:28.458136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:28.458371 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:28.458301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:28.540299 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:28.540264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:28.577356 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:28.577326    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dnjzj" event={"ID":"22778bfc-2110-48e0-85dd-25e5ca26c161","Type":"ContainerStarted","Data":"611f85811d95a38102c9c52e75386bcec7d0fccbbe82be1bd9542b8c3b92e3e1"}
Nov 04 01:02:28.595133 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:28.595078    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dnjzj" podStartSLOduration=2.94262146 podStartE2EDuration="8.595058064s" totalImagesPullingTime="5.652436604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:28.594312214 +0000 UTC m=+8.814016748" watchObservedRunningTime="2026-11-04 01:02:28.595058064 +0000 UTC m=+8.814762599"
Nov 04 01:02:29.458574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:29.458147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:29.458574 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.458291    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:29.458574 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:29.458351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:29.458574 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.458413    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:29.582769 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:29.582736    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerStarted","Data":"d06c2ae4a532a9a522536bc52369e0e323b4575b80c2ba80b0adb96c41d31890"}
Nov 04 01:02:29.781413 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:29.781334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:29.781563 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.781517    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:29.781626 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.781577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:02:37.781559676 +0000 UTC m=+18.001264193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:29.882667 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:29.882625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:29.882836 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:29.882702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:29.882999 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.882966    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:29.883066 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.883050    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:02:37.883031243 +0000 UTC m=+18.102735760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:29.883514 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.883497    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:29.883568 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.883522    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:29.883568 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.883535    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:29.883650 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:29.883588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:02:37.88357156 +0000 UTC m=+18.103276137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:30.460130 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:30.460096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:30.460316 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:30.460241    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:30.644580 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:30.644418    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 04 01:02:31.159868 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.159590    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:02:30.644436821Z","UUID":"a447e22f-6e90-46de-b3a2-32ce2c93923c","Handler":null,"Name":"","Endpoint":""}
Nov 04 01:02:31.161508 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.161486    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 04 01:02:31.161631 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.161517    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 04 01:02:31.458102 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.458025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:31.458244 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:31.458168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:31.458595 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.458576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:31.458701 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:31.458680    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:31.589537 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.589501    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6tjgh" event={"ID":"95384914-9263-45e9-9639-3566767d7559","Type":"ContainerStarted","Data":"32fcff96a47ecebec15f29d97b6e23d545cdf3784424378cfa1b066ac57c8716"}
Nov 04 01:02:31.592602 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.592573    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerStarted","Data":"fecffc0fdc3e7590af5bfc66d3a3d3a3d28d0fd570eaa3df510bbe6df2503f52"}
Nov 04 01:02:31.596492 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.596473    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxdjh" event={"ID":"e8fac715-d046-4c8f-af92-66684a66e0ae","Type":"ContainerStarted","Data":"73cda9874b56ff610eb660e134d010bbe2f2aac1230598775057b7971f296506"}
Nov 04 01:02:31.598910 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.598888    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="37020ed7dbc942d5ec2f8efac53fdd08d980dcefe37367f4c0108470cb245133" exitCode=0
Nov 04 01:02:31.599015 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.598931    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerDied","Data":"37020ed7dbc942d5ec2f8efac53fdd08d980dcefe37367f4c0108470cb245133"}
Nov 04 01:02:31.607626 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.607583    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6tjgh" podStartSLOduration=3.247518878 podStartE2EDuration="11.607567375s" totalImagesPullingTime="8.360048497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:31.607072849 +0000 UTC m=+11.826777595" watchObservedRunningTime="2026-11-04 01:02:31.607567375 +0000 UTC m=+11.827271910"
Nov 04 01:02:31.626579 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.626526    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jxdjh" podStartSLOduration=3.534214293 podStartE2EDuration="11.626510431s" totalImagesPullingTime="8.092296138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:31.626269439 +0000 UTC m=+11.845973973" watchObservedRunningTime="2026-11-04 01:02:31.626510431 +0000 UTC m=+11.846214965"
Nov 04 01:02:31.968637 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.968401    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:31.969142 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:31.969108    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:32.076085 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.076051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:32.277142 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.277067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:32.458122 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.458091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:32.458285 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:32.458244    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:32.604875 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.604842    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerStarted","Data":"190c5f2d3caf702d4b30bde9f650756902c436114df9de076fdb58766ce05155"}
Nov 04 01:02:32.608094 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.608065    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x7xvv" event={"ID":"33e0d88d-b649-469b-905b-25cded0252f9","Type":"ContainerStarted","Data":"09e2fab5fb9817924f615488f33896249ab29498ccf99ebf6e4ee91d6b00d79f"}
Nov 04 01:02:32.625762 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.625718    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podStartSLOduration=3.734322878 podStartE2EDuration="12.625702736s" totalImagesPullingTime="8.891379858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:32.625416019 +0000 UTC m=+12.845120553" watchObservedRunningTime="2026-11-04 01:02:32.625702736 +0000 UTC m=+12.845407269"
Nov 04 01:02:32.642629 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:32.642575    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x7xvv" podStartSLOduration=4.038451988 podStartE2EDuration="12.642559483s" totalImagesPullingTime="8.604107495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:32.642097149 +0000 UTC m=+12.861801681" watchObservedRunningTime="2026-11-04 01:02:32.642559483 +0000 UTC m=+12.862264017"
Nov 04 01:02:33.457843 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:33.457729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:33.457843 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:33.457815    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:33.457843 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:33.457838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:33.458301 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:33.457884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:33.611170 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:33.611087    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vf66l" event={"ID":"552f596f-96f2-4d27-9b5a-1c87f8dc864d","Type":"ContainerStarted","Data":"314a6084f8982246015e83d9d8332a1cc14047f4ecbebc10b5edd4cbcff82873"}
Nov 04 01:02:33.613460 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:33.613431    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="4e140a0e6c87d5267b5bb6fa6ea795d3c1ce7f079fd8c96b65f008bee6b3e29e" exitCode=0
Nov 04 01:02:33.613572 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:33.613504    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerDied","Data":"4e140a0e6c87d5267b5bb6fa6ea795d3c1ce7f079fd8c96b65f008bee6b3e29e"}
Nov 04 01:02:33.630837 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:33.630796    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vf66l" podStartSLOduration=2.63630695 podStartE2EDuration="13.630783461s" totalImagesPullingTime="10.994476511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:33.630508468 +0000 UTC m=+13.850213002" watchObservedRunningTime="2026-11-04 01:02:33.630783461 +0000 UTC m=+13.850487993"
Nov 04 01:02:34.458495 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:34.458457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:34.459090 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:34.458589    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:35.458457 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:35.458422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:35.458618 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:35.458471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:35.458618 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:35.458579    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:35.459313 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:35.458716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:35.622124 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:35.622087    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="553970ce697959ce39971d073d6908e7f9a3864805670c0cade12a1cd376c8de" exitCode=0
Nov 04 01:02:35.622322 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:35.622135    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerDied","Data":"553970ce697959ce39971d073d6908e7f9a3864805670c0cade12a1cd376c8de"}
Nov 04 01:02:35.655030 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:35.654981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:35.870932 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:35.870850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:36.078400 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.078363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:36.307844 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.307549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:36.352326 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.352294    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:36.352482 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.352432    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:02:36.353167 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.353137    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:02:36.457651 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.457614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:36.457802 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:36.457734    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:36.509681 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.509646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:36.730317 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:36.730159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:37.337555 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.337506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:37.457506 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.457481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:37.457673 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.457481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:37.457673 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.457575    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:37.457769 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.457669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:37.536959 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.536927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:37.832436 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.832399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:37.842591 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.842557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:37.842741 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.842719    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:37.842804 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.842792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:02:53.842771961 +0000 UTC m=+34.062476485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:37.944053 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.943808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:37.944132 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.943949    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:37.944132 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.944123    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:37.944697 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.944141    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:37.944697 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:37.944143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:37.944697 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.944205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:02:53.9441823 +0000 UTC m=+34.163886826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:37.944697 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.944221    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:37.944697 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:37.944263    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:02:53.944252615 +0000 UTC m=+34.163957130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:38.160354 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.160305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:38.362382 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.362314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:38.458499 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.458471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:38.458647 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:38.458623    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:38.528221 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.528197    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6tjgh_95384914-9263-45e9-9639-3566767d7559/dns-node-resolver/0.log"
Nov 04 01:02:38.631115 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.631031    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"67d2b7fb11ea20158d825970e6b5bb4e1f90aeb38cfb70049739a47f129bc46f"}
Nov 04 01:02:38.631115 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.631112    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"f6104120961942594a524ab63b87b1aa7ca3b08ea291c16f3d44f5206ba23844"}
Nov 04 01:02:38.631673 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.631156    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"352a5f6dcf9176d2d3c84d9e7ec0eea1a6c814d264380075d205787980f38175"}
Nov 04 01:02:38.631673 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.631200    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"5906fd8f7dc4446256d9e973912a5f1e489bfe355e1af9e8708e744aadd5530e"}
Nov 04 01:02:38.631673 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.631245    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"5290c4d0d52963d926abad3bdbcee643e1740f04bfeb3a700407bda80f9b19a7"}
Nov 04 01:02:38.631836 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.631293    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"502914b1cc1b0aa8a65aa841922aaab95b54d36c9d9b59c85780ffcb0df55945"}
Nov 04 01:02:38.633655 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.633631    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="2b8a6845621ae5374aa2374cde88f25b4b3796afd6cc03098a294260b8324258" exitCode=0
Nov 04 01:02:38.633741 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.633671    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerDied","Data":"2b8a6845621ae5374aa2374cde88f25b4b3796afd6cc03098a294260b8324258"}
Nov 04 01:02:38.959224 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:38.959138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:39.158707 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.158680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:39.458460 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.458367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:39.458639 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.458367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:39.458639 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:39.458478    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:39.458639 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:39.458578    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:39.527961 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.527936    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dnjzj_22778bfc-2110-48e0-85dd-25e5ca26c161/node-ca/0.log"
Nov 04 01:02:39.548785 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.548762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:39.638475 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.638442    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="4c64db676abc87f50f6e524d63b72528c05866dbfa6df43f8b890f85f168ba13" exitCode=0
Nov 04 01:02:39.639027 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.638481    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerDied","Data":"4c64db676abc87f50f6e524d63b72528c05866dbfa6df43f8b890f85f168ba13"}
Nov 04 01:02:39.773003 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:39.772959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:40.154302 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.154276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:40.458235 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.458156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:40.458367 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:40.458252    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:40.507707 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.507677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:40.647163 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.647124    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" event={"ID":"bd730006-869f-4fb3-b626-912e7530f33c","Type":"ContainerStarted","Data":"917b0640db9c384ccd8e2a44a0c426752c7a2405819d8847538db15ba1415694"}
Nov 04 01:02:40.650093 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.650070    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"03d49354933bbbd11d8504c54bafafb82bc594ed937f4e431d6d2d2aad0c885d"}
Nov 04 01:02:40.675449 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.675415    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fjpq6" podStartSLOduration=7.722974292 podStartE2EDuration="20.675403075s" totalImagesPullingTime="12.952428783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:40.674886587 +0000 UTC m=+20.894591121" watchObservedRunningTime="2026-11-04 01:02:40.675403075 +0000 UTC m=+20.895107615"
Nov 04 01:02:40.950028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:40.949926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:41.351026 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:41.350928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:41.458255 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:41.458224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:41.458255 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:41.458246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:41.458423 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:41.458319    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:41.458482 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:41.458436    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:41.551716 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:41.551690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:41.753593 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:41.753511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:41.958881 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:41.958853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:42.153695 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.153604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:42.457890 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.457794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:42.458057 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:42.457914    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:42.657903 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.657867    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" event={"ID":"9e38076f-6754-46f2-b054-f0aa1976319e","Type":"ContainerStarted","Data":"b38b03ff678e266581969fb56509be2745b0d649ac9795b4342360b5a575092b"}
Nov 04 01:02:42.658128 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.658109    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:42.658231 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.658218    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:42.673601 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.673579    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:42.692427 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.692389    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podStartSLOduration=7.472908408 podStartE2EDuration="22.692377186s" totalImagesPullingTime="15.219468778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:42.690549853 +0000 UTC m=+22.910254386" watchObservedRunningTime="2026-11-04 01:02:42.692377186 +0000 UTC m=+22.912081719"
Nov 04 01:02:42.948529 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:42.948459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:43.152577 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.152544    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:43.348721 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.348655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:43.458185 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.458155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:43.458431 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:43.458310    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:43.458431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.458351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:43.458431 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:43.458423    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:43.553346 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.553314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:43.660295 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.660268    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:43.674597 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.674569    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:02:43.756104 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:43.756062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:44.133599 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:44.133570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:44.329549 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:44.329520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:44.458083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:44.458017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:44.458235 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:44.458126    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:44.956445 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:44.956413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:45.355784 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:45.355707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:45.458124 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:45.458101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:45.458269 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:45.458100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:45.458269 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:45.458200    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:45.458269 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:45.458261    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:45.559733 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:45.559699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:45.777030 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:45.776980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:46.156922 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:46.156891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:46.457652 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:46.457570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:46.458136 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:46.457696    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:46.965144 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:46.965111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:47.177626 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.177593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:47.392724 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.392697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:47.457900 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.457872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:47.458247 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:47.457974    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:47.458247 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.458027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:47.458247 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:47.458116    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:47.600276 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.600247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:47.802827 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.802757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:47.998508 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:47.998474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:48.458264 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:48.458232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:48.458655 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:48.458357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:48.810751 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:48.810671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:49.458122 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:49.458092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:49.458300 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:49.458092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:49.458300 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:49.458197    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:49.458300 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:49.458261    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:50.461353 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:50.459333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:50.461353 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:50.459549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:51.415731 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:51.415686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:51.457749 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:51.457716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:51.457749 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:51.457728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:51.457948 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:51.457813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:51.457948 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:51.457917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:51.615226 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:51.615196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:52.296460 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:52.296425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:52.458107 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:52.458078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:52.458328 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:52.458214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:52.493254 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:52.493225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:52.686746 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:52.686718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:53.457562 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.457531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:53.457752 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.457530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:53.457752 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.457650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:53.457752 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.457702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:53.661410 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.661371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:53.858759 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.858684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:53.859172 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.858812    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:53.859172 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.858865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret podName:bc5258b8-2791-49ac-a271-e914db291c6f nodeName:}" failed. No retries permitted until 2026-11-04 01:03:25.858851905 +0000 UTC m=+66.078556416 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret") pod "global-pull-secret-syncer-x7c8v" (UID: "bc5258b8-2791-49ac-a271-e914db291c6f") : object "kube-system"/"original-pull-secret" not registered
Nov 04 01:02:53.859271 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.859208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:53.959467 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.959435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:53.959610 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:53.959484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:53.959610 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.959574    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:53.959610 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.959584    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 04 01:02:53.959610 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.959607    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 04 01:02:53.959739 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.959617    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4shxk for pod openshift-network-diagnostics/network-check-target-58tg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:53.959739 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.959624    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs podName:4b6e932c-bb1d-424e-8dd2-c92797151de6 nodeName:}" failed. No retries permitted until 2026-11-04 01:03:25.959608858 +0000 UTC m=+66.179313382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs") pod "network-metrics-daemon-gfjzp" (UID: "4b6e932c-bb1d-424e-8dd2-c92797151de6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 04 01:02:53.959739 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:53.959659    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk podName:b059c4d3-70d1-4c04-95e9-e1735da565ef nodeName:}" failed. No retries permitted until 2026-11-04 01:03:25.959647935 +0000 UTC m=+66.179352446 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4shxk" (UniqueName: "kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk") pod "network-check-target-58tg6" (UID: "b059c4d3-70d1-4c04-95e9-e1735da565ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 04 01:02:54.057017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:54.056970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:54.265256 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:54.265224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:54.457917 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:54.457888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:54.458104 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:54.458074    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:54.460598 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:54.460570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:54.667452 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:54.667421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:55.457651 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:55.457617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:55.458091 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:55.457713    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:55.458091 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:55.457776    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:55.458091 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:55.457882    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:56.457871 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:56.457843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:56.458283 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:56.457966    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:57.458265 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:57.458235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:57.458644 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:57.458235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:57.458644 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:57.458331    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:57.458644 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:57.458399    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:57.461402 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:57.461369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:57.656829 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:57.656789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:02:58.458506 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:58.458470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:58.458906 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:58.458618    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:02:58.998712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:58.998611    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7c8v"]
Nov 04 01:02:58.998862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:58.998752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:02:58.998922 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:58.998861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:02:59.001416 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:59.001392    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58tg6"]
Nov 04 01:02:59.001693 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:59.001671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:02:59.001813 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:59.001764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:02:59.002156 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:59.002137    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gfjzp"]
Nov 04 01:02:59.002258 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:02:59.002232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:02:59.002355 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:02:59.002329    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:03:00.460904 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:00.460660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:00.461563 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:03:00.461532    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:03:00.461748 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:00.461734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:00.461953 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:03:00.461936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:03:00.657754 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:00.657684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:01.457586 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:01.457554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:01.457755 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:03:01.457651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7c8v" podUID="bc5258b8-2791-49ac-a271-e914db291c6f"
Nov 04 01:03:02.458275 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:02.458241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:02.458669 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:03:02.458356    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gfjzp" podUID="4b6e932c-bb1d-424e-8dd2-c92797151de6"
Nov 04 01:03:02.458669 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:02.458385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:02.458669 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:03:02.458431    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef"
Nov 04 01:03:03.112014 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.111977    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeReady"
Nov 04 01:03:03.112146 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.112106    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 04 01:03:03.159275 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.159250    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z88mh"]
Nov 04 01:03:03.167671 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.167653    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z5jfh"]
Nov 04 01:03:03.167782 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.167763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.175629 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.175606    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 04 01:03:03.175896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.175881    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vklqk\""
Nov 04 01:03:03.176125 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.176110    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 04 01:03:03.178058 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.178038    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z88mh"]
Nov 04 01:03:03.178194 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.178171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.179470 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.179447    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z5jfh"]
Nov 04 01:03:03.183619 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.183601    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lbqr9"]
Nov 04 01:03:03.192305 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.192262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.198313 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.198281    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lbqr9"]
Nov 04 01:03:03.276314 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 04 01:03:03.276314 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 04 01:03:03.276578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276333    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 04 01:03:03.276578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276407    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 04 01:03:03.276578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276455    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7ffl4\""
Nov 04 01:03:03.276578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276461    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 04 01:03:03.276578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276491    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6kj2d\""
Nov 04 01:03:03.276578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.276530    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 04 01:03:03.277553 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.277542    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 04 01:03:03.326872 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.326837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1de55b1-797e-4e61-aeee-624aa4e20ca5-tmp-dir\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.326872 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.326873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf31dccb-516a-4a97-aa86-9277ec17154d-data-volume\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.327147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.326893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de55b1-797e-4e61-aeee-624aa4e20ca5-config-volume\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.327147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.326966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf31dccb-516a-4a97-aa86-9277ec17154d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.327147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f132805a-066a-4d03-bcb3-90c1dc9dc73e-cert\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.327147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.327147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1de55b1-797e-4e61-aeee-624aa4e20ca5-metrics-tls\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.327345 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mwd\" (UniqueName: \"kubernetes.io/projected/f132805a-066a-4d03-bcb3-90c1dc9dc73e-kube-api-access-t9mwd\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.327345 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxcmd\" (UniqueName: \"kubernetes.io/projected/b1de55b1-797e-4e61-aeee-624aa4e20ca5-kube-api-access-nxcmd\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.327345 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf31dccb-516a-4a97-aa86-9277ec17154d-crio-socket\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.327345 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.327245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprmj\" (UniqueName: \"kubernetes.io/projected/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-api-access-cprmj\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.427858 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.427817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1de55b1-797e-4e61-aeee-624aa4e20ca5-tmp-dir\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.428028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.427882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf31dccb-516a-4a97-aa86-9277ec17154d-data-volume\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.427904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de55b1-797e-4e61-aeee-624aa4e20ca5-config-volume\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.428028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.427922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf31dccb-516a-4a97-aa86-9277ec17154d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.427942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f132805a-066a-4d03-bcb3-90c1dc9dc73e-cert\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.428028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.427973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428028 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428023    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1de55b1-797e-4e61-aeee-624aa4e20ca5-tmp-dir\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.428332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1de55b1-797e-4e61-aeee-624aa4e20ca5-metrics-tls\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.428332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mwd\" (UniqueName: \"kubernetes.io/projected/f132805a-066a-4d03-bcb3-90c1dc9dc73e-kube-api-access-t9mwd\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.428332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcmd\" (UniqueName: \"kubernetes.io/projected/b1de55b1-797e-4e61-aeee-624aa4e20ca5-kube-api-access-nxcmd\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.428332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf31dccb-516a-4a97-aa86-9277ec17154d-crio-socket\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428332 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprmj\" (UniqueName: \"kubernetes.io/projected/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-api-access-cprmj\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428569 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428346    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf31dccb-516a-4a97-aa86-9277ec17154d-data-volume\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428569 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf31dccb-516a-4a97-aa86-9277ec17154d-crio-socket\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.428671 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de55b1-797e-4e61-aeee-624aa4e20ca5-config-volume\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.428942 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.428925    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.431901 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.431873    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1de55b1-797e-4e61-aeee-624aa4e20ca5-metrics-tls\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.432002 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.431877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf31dccb-516a-4a97-aa86-9277ec17154d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.432002 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.431962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f132805a-066a-4d03-bcb3-90c1dc9dc73e-cert\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.440537 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.440503    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprmj\" (UniqueName: \"kubernetes.io/projected/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-api-access-cprmj\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.440644 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.440559    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcmd\" (UniqueName: \"kubernetes.io/projected/b1de55b1-797e-4e61-aeee-624aa4e20ca5-kube-api-access-nxcmd\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.440739 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.440721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mwd\" (UniqueName: \"kubernetes.io/projected/f132805a-066a-4d03-bcb3-90c1dc9dc73e-kube-api-access-t9mwd\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.457673 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.457627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:03.462142 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.462108    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 04 01:03:03.462700 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.462169    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9c9zb\""
Nov 04 01:03:03.493046 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.493028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:03.501533 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.501518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:03.528108 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.528086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:03.830777 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.830700    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z5jfh"]
Nov 04 01:03:03.833791 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.833718    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z88mh"]
Nov 04 01:03:03.834842 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:03.834818    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lbqr9"]
Nov 04 01:03:03.835410 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:03.835383    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf132805a_066a_4d03_bcb3_90c1dc9dc73e.slice/crio-89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164 WatchSource:0}: Error finding container 89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164: Status 404 returned error can't find the container with id 89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164
Nov 04 01:03:03.837604 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:03.837579    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1de55b1_797e_4e61_aeee_624aa4e20ca5.slice/crio-0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a WatchSource:0}: Error finding container 0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a: Status 404 returned error can't find the container with id 0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a
Nov 04 01:03:03.838474 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:03.838455    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf31dccb_516a_4a97_aa86_9277ec17154d.slice/crio-649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96 WatchSource:0}: Error finding container 649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96: Status 404 returned error can't find the container with id 649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96
Nov 04 01:03:04.461609 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.461581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:04.461844 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.461610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:04.466122 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.465984    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 04 01:03:04.466122 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.466055    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 04 01:03:04.467505 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.467484    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 04 01:03:04.467596 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.467518    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5jp5j\""
Nov 04 01:03:04.467596 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.467531    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-k9wxw\""
Nov 04 01:03:04.703489 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.703454    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbqr9" event={"ID":"bf31dccb-516a-4a97-aa86-9277ec17154d","Type":"ContainerStarted","Data":"d0ab76f49c1f601935b35ff4b3eb4ba6cb00fc765085bf9406e1cff71b196045"}
Nov 04 01:03:04.703644 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.703519    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbqr9" event={"ID":"bf31dccb-516a-4a97-aa86-9277ec17154d","Type":"ContainerStarted","Data":"649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96"}
Nov 04 01:03:04.704759 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.704731    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z5jfh" event={"ID":"f132805a-066a-4d03-bcb3-90c1dc9dc73e","Type":"ContainerStarted","Data":"89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164"}
Nov 04 01:03:04.705955 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:04.705921    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z88mh" event={"ID":"b1de55b1-797e-4e61-aeee-624aa4e20ca5","Type":"ContainerStarted","Data":"0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a"}
Nov 04 01:03:05.669914 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.669878    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f8d87f9b-d526x"]
Nov 04 01:03:05.673264 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.673241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.679184 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.679162    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 04 01:03:05.682401 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.682016    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 04 01:03:05.682401 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.682047    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 04 01:03:05.682401 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.682131    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 04 01:03:05.683878 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.683655    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 04 01:03:05.683878 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.683776    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x6b8r\""
Nov 04 01:03:05.683878 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.683847    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 04 01:03:05.684675 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.684654    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f8d87f9b-d526x"]
Nov 04 01:03:05.709643 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.709618    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbqr9" event={"ID":"bf31dccb-516a-4a97-aa86-9277ec17154d","Type":"ContainerStarted","Data":"7742a0c8aa76f1e0b3448274b02c55cde82c259a3668af7e846b0606e250e57a"}
Nov 04 01:03:05.779383 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.779356    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 04 01:03:05.845325 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.845325 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.845567 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.845567 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.845567 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.845567 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.845753 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.845596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.946823 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.946754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.946823 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.946790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.946823 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.946817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947048 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.946999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947084 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947084 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947154 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947476 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947457    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947671 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947737 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.947782 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.947743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.950364 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.950346    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.950464 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.950349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.959501 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.959475    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:05.988512 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:05.988489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:06.265646 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:06.265568    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f8d87f9b-d526x"]
Nov 04 01:03:06.275167 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:06.275125    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472ca873_6ba4_403c_9a6c_2c330a7ac131.slice/crio-152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da WatchSource:0}: Error finding container 152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da: Status 404 returned error can't find the container with id 152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da
Nov 04 01:03:06.417862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:06.417827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:06.614210 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:06.614135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:06.713761 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:06.713686    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8d87f9b-d526x" event={"ID":"472ca873-6ba4-403c-9a6c-2c330a7ac131","Type":"ContainerStarted","Data":"152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da"}
Nov 04 01:03:07.404948 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.404698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:07.465059 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.465029    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"]
Nov 04 01:03:07.468146 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.468124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.472274 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.472203    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 04 01:03:07.472274 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.472243    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 04 01:03:07.472274 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.472250    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kzl6m\""
Nov 04 01:03:07.473189 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.473146    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 04 01:03:07.473316 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.473228    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 04 01:03:07.473316 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.473310    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 04 01:03:07.477897 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.477861    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"]
Nov 04 01:03:07.632474 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.632439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:07.662914 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.662838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.663068 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.662916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.663068 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.662998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8mm\" (UniqueName: \"kubernetes.io/projected/b5df0493-dbfc-4398-94fa-2d5ec7676197-kube-api-access-7c8mm\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.663068 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.663060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5df0493-dbfc-4398-94fa-2d5ec7676197-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.718954 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.718920    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z88mh" event={"ID":"b1de55b1-797e-4e61-aeee-624aa4e20ca5","Type":"ContainerStarted","Data":"a3dc65d64a16a9d0fc2422de3bd93d57b18e4d3d0b695b4a1942d8919efca2f0"}
Nov 04 01:03:07.719733 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.718983    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z88mh" event={"ID":"b1de55b1-797e-4e61-aeee-624aa4e20ca5","Type":"ContainerStarted","Data":"7712ac76d721a3f9f0f022d18434041e5fd0fb907619f0bc8123cab47daf74c1"}
Nov 04 01:03:07.719733 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.719043    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:07.721739 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.721713    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbqr9" event={"ID":"bf31dccb-516a-4a97-aa86-9277ec17154d","Type":"ContainerStarted","Data":"3cb6d1cd38f4e28fea12d7ebeb46acdce82d130e7fc042249e7e6d4378d81700"}
Nov 04 01:03:07.740677 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.740633    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-z88mh" podStartSLOduration=1.857527017 podStartE2EDuration="4.740621037s" totalImagesPullingTime="2.88309402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:07.738711132 +0000 UTC m=+47.958415677" watchObservedRunningTime="2026-11-04 01:03:07.740621037 +0000 UTC m=+47.960325600"
Nov 04 01:03:07.759824 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.759774    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lbqr9" podStartSLOduration=1.7030733630000001 podStartE2EDuration="4.75975736s" totalImagesPullingTime="3.056683997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:03:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:07.759625609 +0000 UTC m=+47.979330143" watchObservedRunningTime="2026-11-04 01:03:07.75975736 +0000 UTC m=+47.979461893"
Nov 04 01:03:07.763838 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.763801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.763948 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.763859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8mm\" (UniqueName: \"kubernetes.io/projected/b5df0493-dbfc-4398-94fa-2d5ec7676197-kube-api-access-7c8mm\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.763948 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.763900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5df0493-dbfc-4398-94fa-2d5ec7676197-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.764037 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.763946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.764828 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.764808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5df0493-dbfc-4398-94fa-2d5ec7676197-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.767801 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.767776    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.767905 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.767784    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.778151 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.778126    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8mm\" (UniqueName: \"kubernetes.io/projected/b5df0493-dbfc-4398-94fa-2d5ec7676197-kube-api-access-7c8mm\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.783916 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.783894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:07.828290 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:07.828258    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:08.067361 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:08.066896    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"]
Nov 04 01:03:08.070105 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:08.070066    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5df0493_dbfc_4398_94fa_2d5ec7676197.slice/crio-e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462 WatchSource:0}: Error finding container e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462: Status 404 returned error can't find the container with id e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462
Nov 04 01:03:08.726703 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:08.726667    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z5jfh" event={"ID":"f132805a-066a-4d03-bcb3-90c1dc9dc73e","Type":"ContainerStarted","Data":"8a21a94d067a27343bc3b73b23881f6f0151c1a9a788ae97eb253a1f2b018c7d"}
Nov 04 01:03:08.728007 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:08.727965    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn" event={"ID":"b5df0493-dbfc-4398-94fa-2d5ec7676197","Type":"ContainerStarted","Data":"e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462"}
Nov 04 01:03:08.746135 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:08.746085    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z5jfh" podStartSLOduration=1.18526454 podStartE2EDuration="5.746066986s" totalImagesPullingTime="4.560802446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:08.745452071 +0000 UTC m=+48.965156603" watchObservedRunningTime="2026-11-04 01:03:08.746066986 +0000 UTC m=+48.965771520"
Nov 04 01:03:09.231749 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:09.231712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:09.432865 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:09.432824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:10.237554 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.237517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:10.433417 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.433384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:10.734853 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.734772    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8d87f9b-d526x" event={"ID":"472ca873-6ba4-403c-9a6c-2c330a7ac131","Type":"ContainerStarted","Data":"d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c"}
Nov 04 01:03:10.736499 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.736472    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn" event={"ID":"b5df0493-dbfc-4398-94fa-2d5ec7676197","Type":"ContainerStarted","Data":"a8686b23dbd5b77b7e891bbdb354db9187d3696936e94319302fd48e857bbd15"}
Nov 04 01:03:10.736630 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.736514    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn" event={"ID":"b5df0493-dbfc-4398-94fa-2d5ec7676197","Type":"ContainerStarted","Data":"ee67a54c868337dc4f8bc8ae91bf2859b3e688e6fe26c304366a7b4f28407ddc"}
Nov 04 01:03:10.755223 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.755179    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67f8d87f9b-d526x" podStartSLOduration=1.985759839 podStartE2EDuration="5.755165174s" totalImagesPullingTime="3.769405335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:10.754762065 +0000 UTC m=+50.974466597" watchObservedRunningTime="2026-11-04 01:03:10.755165174 +0000 UTC m=+50.974869706"
Nov 04 01:03:10.774715 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:10.774669    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn" podStartSLOduration=1.849087412 podStartE2EDuration="3.774658265s" totalImagesPullingTime="1.925570853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:10.77446267 +0000 UTC m=+50.994167215" watchObservedRunningTime="2026-11-04 01:03:10.774658265 +0000 UTC m=+50.994362798"
Nov 04 01:03:11.200816 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:11.200785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:11.400049 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:11.400008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:12.692374 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.692342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:12.900178 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.900149    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-29mzn"]
Nov 04 01:03:12.903379 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.903359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:12.910605 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.910579    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"]
Nov 04 01:03:12.913616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.913595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:12.947686 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.947630    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"]
Nov 04 01:03:12.952975 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.952956    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 04 01:03:12.953168 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.953123    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 04 01:03:12.953238 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.953211    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 04 01:03:12.953703 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.953689    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gj86p\""
Nov 04 01:03:12.953765 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.953707    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6wds\""
Nov 04 01:03:12.953765 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.953729    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 04 01:03:12.965488 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.965431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:12.980865 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.980832    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 04 01:03:12.981059 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:12.981043    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 04 01:03:13.007393 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.007488 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-tls\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007488 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-metrics-client-ca\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007569 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-root\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007569 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007569 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-wtmp\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007656 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-accelerators-collector-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007707 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.007742 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-sys\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007772 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.007804 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdkv\" (UniqueName: \"kubernetes.io/projected/513cdfcf-ac71-4d24-aca9-1c81edb6f803-kube-api-access-gmdkv\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007833 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8qjz\" (UniqueName: \"kubernetes.io/projected/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-api-access-v8qjz\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.007871 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9693b4e7-8f19-4e24-aac8-4fb293180bae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.007904 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-textfile\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.007904 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.007901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.109220 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-root\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109220 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109398 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-wtmp\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109398 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-accelerators-collector-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109398 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109303    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-root\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109398 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.109398 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-sys\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdkv\" (UniqueName: \"kubernetes.io/projected/513cdfcf-ac71-4d24-aca9-1c81edb6f803-kube-api-access-gmdkv\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109444    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-wtmp\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qjz\" (UniqueName: \"kubernetes.io/projected/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-api-access-v8qjz\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109492    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-sys\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9693b4e7-8f19-4e24-aac8-4fb293180bae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-textfile\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.109616 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.110116 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.110116 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-tls\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.110116 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-metrics-client-ca\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.110116 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.109873    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-textfile\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.110341 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.110196    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.110341 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.110321    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-accelerators-collector-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.110457 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.110323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-metrics-client-ca\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.110501 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.110452    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.110634 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.110542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9693b4e7-8f19-4e24-aac8-4fb293180bae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.112289 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.112265    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.112708 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.112685    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.112798 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.112694    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.112798 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.112774    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-tls\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.124123 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.124093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdkv\" (UniqueName: \"kubernetes.io/projected/513cdfcf-ac71-4d24-aca9-1c81edb6f803-kube-api-access-gmdkv\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.124732 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.124710    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qjz\" (UniqueName: \"kubernetes.io/projected/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-api-access-v8qjz\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.174597 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.174563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:13.219060 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.218998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:13.228147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.228126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:13.229773 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:13.229752    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513cdfcf_ac71_4d24_aca9_1c81edb6f803.slice/crio-b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73 WatchSource:0}: Error finding container b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73: Status 404 returned error can't find the container with id b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73
Nov 04 01:03:13.399105 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.399075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:13.513346 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.513184    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"]
Nov 04 01:03:13.515713 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:13.515685    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9693b4e7_8f19_4e24_aac8_4fb293180bae.slice/crio-5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258 WatchSource:0}: Error finding container 5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258: Status 404 returned error can't find the container with id 5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258
Nov 04 01:03:13.618713 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.618658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:13.750471 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.750436    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29mzn" event={"ID":"513cdfcf-ac71-4d24-aca9-1c81edb6f803","Type":"ContainerStarted","Data":"b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73"}
Nov 04 01:03:13.751482 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.751462    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk" event={"ID":"9693b4e7-8f19-4e24-aac8-4fb293180bae","Type":"ContainerStarted","Data":"5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258"}
Nov 04 01:03:13.991413 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.991378    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 04 01:03:13.995530 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:13.995508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002757    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002791    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002805    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002819    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002757    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002842    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lkq85\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002844    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002875    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 04 01:03:14.002896 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002901    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 04 01:03:14.003419 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.002909    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 04 01:03:14.015328 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015304    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 04 01:03:14.015596 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.015715 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.015715 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.015841 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.015897 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.015948 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.015948 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.016074 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.015953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.016074 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.016046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.016167 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.016106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.016167 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.016125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.016167 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.016142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.016167 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.016159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.023020 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.022056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:14.117308 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117485 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117485 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117485 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117485 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117485 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117485 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117696 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117696 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117696 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117696 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117696 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.117696 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.117656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.118062 ip-10-0-140-186 kubenswrapper[2579]: E1104 01:03:14.118043    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle podName:50f13cde-bfd4-4cd7-87f4-7238e4634bf6 nodeName:}" failed. No retries permitted until 2026-11-04 01:03:14.618020026 +0000 UTC m=+54.837724552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6") : configmap references non-existent config key: ca-bundle.crt
Nov 04 01:03:14.118463 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.118434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.121204 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.120737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.121204 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.120910    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.121204 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.121185    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.121790 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.121765    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.122639 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.122059    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.122639 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.122314    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.123109 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.123089    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.123238 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.123218    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.123306 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.123287    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.123341 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.123318    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.133712 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.133691    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.260744 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.260658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:14.623230 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.623180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.624063 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.624015    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:14.756891 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.756860    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29mzn" event={"ID":"513cdfcf-ac71-4d24-aca9-1c81edb6f803","Type":"ContainerStarted","Data":"8dbdb0c7914e22c26d48bcb67f5397f4c97bd8bc60101e30f66bf04e40067c14"}
Nov 04 01:03:14.922490 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:14.922413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:15.210588 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.210567    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 04 01:03:15.211927 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:15.211892    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f13cde_bfd4_4cd7_87f4_7238e4634bf6.slice/crio-dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf WatchSource:0}: Error finding container dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf: Status 404 returned error can't find the container with id dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf
Nov 04 01:03:15.674425 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.674400    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:15.761713 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.761674    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk" event={"ID":"9693b4e7-8f19-4e24-aac8-4fb293180bae","Type":"ContainerStarted","Data":"297f5a3cec5c490a0f9e614101ee4094de8f51c7238f95417273d27bdb6b0d60"}
Nov 04 01:03:15.762160 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.761743    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk" event={"ID":"9693b4e7-8f19-4e24-aac8-4fb293180bae","Type":"ContainerStarted","Data":"d63dc9ed5415f18949f0bec79c21ef5acd69964873a762ced1f3caa2ddf45684"}
Nov 04 01:03:15.762160 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.761774    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk" event={"ID":"9693b4e7-8f19-4e24-aac8-4fb293180bae","Type":"ContainerStarted","Data":"06b4f259a399768fc19e867bb60a6c8a5fa752cde02f65852a4a59730a76697e"}
Nov 04 01:03:15.763141 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.763116    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf"}
Nov 04 01:03:15.764792 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.764761    2579 generic.go:356] "Generic (PLEG): container finished" podID="513cdfcf-ac71-4d24-aca9-1c81edb6f803" containerID="8dbdb0c7914e22c26d48bcb67f5397f4c97bd8bc60101e30f66bf04e40067c14" exitCode=0
Nov 04 01:03:15.764869 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.764811    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29mzn" event={"ID":"513cdfcf-ac71-4d24-aca9-1c81edb6f803","Type":"ContainerDied","Data":"8dbdb0c7914e22c26d48bcb67f5397f4c97bd8bc60101e30f66bf04e40067c14"}
Nov 04 01:03:15.784162 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.784110    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk" podStartSLOduration=2.133314899 podStartE2EDuration="3.78409003s" totalImagesPullingTime="1.650775131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:15.782898622 +0000 UTC m=+56.002603155" watchObservedRunningTime="2026-11-04 01:03:15.78409003 +0000 UTC m=+56.003794565"
Nov 04 01:03:15.970442 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.970367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:15.989147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.989122    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:15.989279 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.989178    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:15.994182 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:15.994161    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:16.182039 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.179629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:16.394646 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.394598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:16.769779 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.769698    2579 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49" exitCode=0
Nov 04 01:03:16.770234 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.769779    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49"}
Nov 04 01:03:16.772223 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.772189    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29mzn" event={"ID":"513cdfcf-ac71-4d24-aca9-1c81edb6f803","Type":"ContainerStarted","Data":"74982eedf85dbd43b13ddfef6c7c956384cfaec59cbabd84d1654c67f5d6fed4"}
Nov 04 01:03:16.772344 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.772251    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29mzn" event={"ID":"513cdfcf-ac71-4d24-aca9-1c81edb6f803","Type":"ContainerStarted","Data":"b8dd590fca470f42b0b09c3f9d18590cd4f76f411da15875f75c38b6f6f03159"}
Nov 04 01:03:16.776346 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.776328    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:16.826739 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:16.826696    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-29mzn" podStartSLOduration=3.304978751 podStartE2EDuration="4.826682894s" totalImagesPullingTime="1.447006756s" totalInitContainerRuntime="74.697387ms" isStatefulPod=true podCreationTimestamp="2026-11-04 01:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:16.825476649 +0000 UTC m=+57.045181182" watchObservedRunningTime="2026-11-04 01:03:16.826682894 +0000 UTC m=+57.046387461"
Nov 04 01:03:17.289593 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.289564    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"]
Nov 04 01:03:17.292653 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.292633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.298048 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.298028    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4sne5ifg6e6mn\""
Nov 04 01:03:17.298147 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.298028    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 04 01:03:17.298337 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.298319    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-grc8c\""
Nov 04 01:03:17.298503 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.298481    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 04 01:03:17.298562 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.298510    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 04 01:03:17.298730 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.298569    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 04 01:03:17.305503 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.305481    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"]
Nov 04 01:03:17.348499 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a2e54ab-acb8-4611-bf2d-534053133f77-audit-log\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.348687 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-metrics-server-audit-profiles\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.348687 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-client-certs\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.348687 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-tls\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.348687 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.348835 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-client-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.348835 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.348789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlwf\" (UniqueName: \"kubernetes.io/projected/1a2e54ab-acb8-4611-bf2d-534053133f77-kube-api-access-wrlwf\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449667 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-client-certs\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449667 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-tls\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449892 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449892 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-client-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449892 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlwf\" (UniqueName: \"kubernetes.io/projected/1a2e54ab-acb8-4611-bf2d-534053133f77-kube-api-access-wrlwf\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449892 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a2e54ab-acb8-4611-bf2d-534053133f77-audit-log\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.449892 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.449853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-metrics-server-audit-profiles\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.450287 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.450258    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a2e54ab-acb8-4611-bf2d-534053133f77-audit-log\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.450470 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.450452    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.450852 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.450827    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-metrics-server-audit-profiles\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.452124 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.452096    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-client-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.452213 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.452149    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-tls\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.452299 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.452279    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-client-certs\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.461957 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.461936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlwf\" (UniqueName: \"kubernetes.io/projected/1a2e54ab-acb8-4611-bf2d-534053133f77-kube-api-access-wrlwf\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.602961 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.602872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:17.724272 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.724235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:17.730706 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.730681    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:17.893792 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.893762    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c94f7f5d6-x8mxr"]
Nov 04 01:03:17.898277 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.898248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.909156 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.909122    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c94f7f5d6-x8mxr"]
Nov 04 01:03:17.913963 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.913938    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 04 01:03:17.916608 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.916582    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"]
Nov 04 01:03:17.954589 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954589 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954767 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954767 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954767 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954767 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954915 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.954915 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.954895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:17.955403 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:17.955361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056083 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.055778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056500 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.056361    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056659 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.056635    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056918 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.056899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.056955 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.056899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.057359 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.057332    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.058178 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.058153    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.058379 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.058362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.067811 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.067789    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.100792 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.100761    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b6c7df88-lkrmc"]
Nov 04 01:03:18.105456 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.105438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.109632 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.109611    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 04 01:03:18.114519 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.114501    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 04 01:03:18.120141 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.120121    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b6c7df88-lkrmc"]
Nov 04 01:03:18.150862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.150782    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 04 01:03:18.150862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.150814    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 04 01:03:18.150862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.150817    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 04 01:03:18.150862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.150822    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 04 01:03:18.150862 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.150872    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-66v8h\""
Nov 04 01:03:18.156275 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156395 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156395 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-serving-certs-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156497 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156557 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156612 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zsw8\" (UniqueName: \"kubernetes.io/projected/77b967b9-f2b0-4b23-9484-038568e3a0ea-kube-api-access-5zsw8\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156664 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-metrics-client-ca\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.156664 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.156645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-federate-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.169014 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.168966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:18.211845 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.211821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-serving-certs-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zsw8\" (UniqueName: \"kubernetes.io/projected/77b967b9-f2b0-4b23-9484-038568e3a0ea-kube-api-access-5zsw8\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-metrics-client-ca\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-federate-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.257632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259006 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.258619    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.259786 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.259730    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-metrics-client-ca\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.260433 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.260405    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-serving-certs-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.261329 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.261278    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.264247 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.263413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-federate-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.264247 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.263445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.264247 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.263715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.279035 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.278925    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zsw8\" (UniqueName: \"kubernetes.io/projected/77b967b9-f2b0-4b23-9484-038568e3a0ea-kube-api-access-5zsw8\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.407809 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.407712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:18.421690 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.421666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:18.492358 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.492334    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c94f7f5d6-x8mxr"]
Nov 04 01:03:18.496519 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:18.496484    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd816f5_e80b_4a1b_9b57_feadd931954f.slice/crio-9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212 WatchSource:0}: Error finding container 9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212: Status 404 returned error can't find the container with id 9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212
Nov 04 01:03:18.610752 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.610718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:18.707259 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.707182    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b6c7df88-lkrmc"]
Nov 04 01:03:18.714824 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:18.714790    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b967b9_f2b0_4b23_9484_038568e3a0ea.slice/crio-245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52 WatchSource:0}: Error finding container 245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52: Status 404 returned error can't find the container with id 245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52
Nov 04 01:03:18.781154 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.781115    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc" event={"ID":"77b967b9-f2b0-4b23-9484-038568e3a0ea","Type":"ContainerStarted","Data":"245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52"}
Nov 04 01:03:18.783523 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.783484    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c94f7f5d6-x8mxr" event={"ID":"7fd816f5-e80b-4a1b-9b57-feadd931954f","Type":"ContainerStarted","Data":"48fb538b782a141a5419f32c9e8fe764ae86b73edfb23ccb9c7571afbf33202f"}
Nov 04 01:03:18.783667 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.783537    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c94f7f5d6-x8mxr" event={"ID":"7fd816f5-e80b-4a1b-9b57-feadd931954f","Type":"ContainerStarted","Data":"9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212"}
Nov 04 01:03:18.785260 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.785236    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" event={"ID":"1a2e54ab-acb8-4611-bf2d-534053133f77","Type":"ContainerStarted","Data":"6293a177e01cda80b75f7a92986e2a5d86aa9a598c15c0184264a7589b194f01"}
Nov 04 01:03:18.805566 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:18.805519    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c94f7f5d6-x8mxr" podStartSLOduration=1.80550547 podStartE2EDuration="1.80550547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:18.803747408 +0000 UTC m=+59.023451946" watchObservedRunningTime="2026-11-04 01:03:18.80550547 +0000 UTC m=+59.025209983"
Nov 04 01:03:19.189838 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.189812    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:03:19.194417 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.194393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.206886    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207136    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207222    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207286    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207410    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9426ur6dsotl2\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207532    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207609    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207696    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.207874    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 04 01:03:19.210206 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.208953    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 04 01:03:19.238040 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.235846    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 04 01:03:19.238201 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.237980    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:03:19.240263 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.240239    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.271984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.272431 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.273217 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.273217 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.273217 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.273217 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.272182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.299220 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.299191    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 04 01:03:19.299339 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.299321    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 04 01:03:19.299434 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.299420    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-48ts5\""
Nov 04 01:03:19.299578 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.299565    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 04 01:03:19.319749 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.319722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:19.373162 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373162 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373350 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373350 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373459 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373459 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373459 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373600 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373477    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373600 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373600 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373600 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373600 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373600 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.373873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.374120 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.373943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.374120 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.374042    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.374120 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.374037    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.374428 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.374398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.375835 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.375805    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.377087 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.377065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.377191 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.377102    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.377550 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.377524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.377943 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.377919    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379104 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.378475    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379104 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.378518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379249 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.378528    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379249 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.378744    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379249 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.378839    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379249 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.379165    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379249 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.378952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379791 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.379774    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.379953 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.379936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.388318 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.388291    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.544535 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.544457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:19.567752 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.567723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:19.798675 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.798451    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321"}
Nov 04 01:03:19.798675 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.798534    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e"}
Nov 04 01:03:19.798675 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.798565    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe"}
Nov 04 01:03:19.799076 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.798603    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca"}
Nov 04 01:03:19.799076 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.799023    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc"}
Nov 04 01:03:19.858415 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:19.858121    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:03:19.860168 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:19.860142    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1430b29f_afb4_4814_ab0b_d31458805413.slice/crio-500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967 WatchSource:0}: Error finding container 500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967: Status 404 returned error can't find the container with id 500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967
Nov 04 01:03:20.803961 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:20.803731    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" event={"ID":"1a2e54ab-acb8-4611-bf2d-534053133f77","Type":"ContainerStarted","Data":"277377df435b3a2c3074d418d2ff01b1b30f5320e301cdcd76f9450a43c2bc49"}
Nov 04 01:03:20.805472 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:20.805445    2579 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="a59aff40f277c5ad13ac6a639828eb521384f3b8dbbc403bb35b6bc540403324" exitCode=0
Nov 04 01:03:20.805562 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:20.805529    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"a59aff40f277c5ad13ac6a639828eb521384f3b8dbbc403bb35b6bc540403324"}
Nov 04 01:03:20.805623 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:20.805609    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967"}
Nov 04 01:03:20.829571 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:20.829482    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podStartSLOduration=1.531029188 podStartE2EDuration="3.829462089s" totalImagesPullingTime="2.298432901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:20.82751612 +0000 UTC m=+61.047220655" watchObservedRunningTime="2026-11-04 01:03:20.829462089 +0000 UTC m=+61.049166623"
Nov 04 01:03:21.411680 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.411646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:21.611848 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.611769    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:21.812789 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.812756    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerStarted","Data":"1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4"}
Nov 04 01:03:21.814832 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.814803    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc" event={"ID":"77b967b9-f2b0-4b23-9484-038568e3a0ea","Type":"ContainerStarted","Data":"5ee75c16291b5820839119a72e64bff0b0f84060d2ef18538d2a694d50d126cd"}
Nov 04 01:03:21.814932 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.814858    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc" event={"ID":"77b967b9-f2b0-4b23-9484-038568e3a0ea","Type":"ContainerStarted","Data":"6deff4219b7ff6d20d5c99e1b074349a09ce821641d7983e71b97cd953fc4071"}
Nov 04 01:03:21.814932 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.814885    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc" event={"ID":"77b967b9-f2b0-4b23-9484-038568e3a0ea","Type":"ContainerStarted","Data":"986a38f972516b43e162da3062d27567efc0b217598c650ce08ac6bb1eeff891"}
Nov 04 01:03:21.850570 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.850523    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.911158098 podStartE2EDuration="8.850509073s" totalImagesPullingTime="4.927262145s" totalInitContainerRuntime="12.08883ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:21.848246593 +0000 UTC m=+62.067951127" watchObservedRunningTime="2026-11-04 01:03:21.850509073 +0000 UTC m=+62.070213606"
Nov 04 01:03:21.876250 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:21.876188    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc" podStartSLOduration=1.381715078 podStartE2EDuration="3.876170361s" totalImagesPullingTime="2.494455283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:21.874444957 +0000 UTC m=+62.094149492" watchObservedRunningTime="2026-11-04 01:03:21.876170361 +0000 UTC m=+62.095874897"
Nov 04 01:03:22.506054 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.506018    2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c94f7f5d6-x8mxr"]
Nov 04 01:03:22.535659 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.535631    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c845bd84f-szzk2"]
Nov 04 01:03:22.539007 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.538963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.551331 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.551304    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c845bd84f-szzk2"]
Nov 04 01:03:22.607873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.607831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.607873 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.607878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.608106 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.608084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.608184 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.608153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.608262 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.608199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.608262 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.608250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.608346 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.608295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.608396 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.608375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.613519 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.613486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:22.709514 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709698 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709698 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709698 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709698 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709698 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709938 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.709938 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.709750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.710375 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.710355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.710468 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.710434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.711475 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.710910    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.711475 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.710959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.711475 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.711441    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.712942 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.712903    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.713801 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.713782    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.725781 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.723654    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:22.812928 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.812849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:22.856996 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:22.856971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:23.136052 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:23.136029    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c845bd84f-szzk2"]
Nov 04 01:03:23.138313 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:23.138285    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1b8f08_7323_48fd_9312_da6169e52c2a.slice/crio-98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305 WatchSource:0}: Error finding container 98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305: Status 404 returned error can't find the container with id 98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305
Nov 04 01:03:23.825186 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:23.825147    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c845bd84f-szzk2" event={"ID":"5e1b8f08-7323-48fd-9312-da6169e52c2a","Type":"ContainerStarted","Data":"d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3"}
Nov 04 01:03:23.825186 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:23.825194    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c845bd84f-szzk2" event={"ID":"5e1b8f08-7323-48fd-9312-da6169e52c2a","Type":"ContainerStarted","Data":"98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305"}
Nov 04 01:03:23.848503 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:23.848455    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c845bd84f-szzk2" podStartSLOduration=1.8484418950000001 podStartE2EDuration="1.848441895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:23.846915267 +0000 UTC m=+64.066619813" watchObservedRunningTime="2026-11-04 01:03:23.848441895 +0000 UTC m=+64.068146427"
Nov 04 01:03:24.813938 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:24.813908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:25.010890 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.010818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:25.836634 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.836598    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"3edef76f89e8de9ebd150ee4476f4f2869a08e4f708725e3e4c395dbc0951314"}
Nov 04 01:03:25.836781 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.836705    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"b483a4ae86d16ba0260df6d7468b828c6ea67dcfe7ee7f5609580a839a10715c"}
Nov 04 01:03:25.949125 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.949040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:25.952266 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.952245    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 04 01:03:25.961392 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.961368    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:25.995934 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:25.995912    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9c9zb\""
Nov 04 01:03:26.004017 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.003981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:26.049848 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.049814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:26.050441 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.049906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:26.053231 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.053212    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 04 01:03:26.053327 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.053312    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 04 01:03:26.063187 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.063161    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:26.063303 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.063173    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 04 01:03:26.073456 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.073433    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:26.079092 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.079021    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5jp5j\""
Nov 04 01:03:26.081455 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.081433    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-k9wxw\""
Nov 04 01:03:26.086579 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.086566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:26.089802 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.089782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:26.273749 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.273726    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7c8v"]
Nov 04 01:03:26.275925 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:26.275897    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5258b8_2791_49ac_a271_e914db291c6f.slice/crio-4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf WatchSource:0}: Error finding container 4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf: Status 404 returned error can't find the container with id 4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf
Nov 04 01:03:26.363212 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.363191    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58tg6"]
Nov 04 01:03:26.365583 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.365564    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gfjzp"]
Nov 04 01:03:26.365998 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:26.365928    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb059c4d3_70d1_4c04_95e9_e1735da565ef.slice/crio-6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524 WatchSource:0}: Error finding container 6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524: Status 404 returned error can't find the container with id 6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524
Nov 04 01:03:26.368020 ip-10-0-140-186 kubenswrapper[2579]: W1104 01:03:26.367985    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6e932c_bb1d_424e_8dd2_c92797151de6.slice/crio-b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943 WatchSource:0}: Error finding container b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943: Status 404 returned error can't find the container with id b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943
Nov 04 01:03:26.815027 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.814979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:26.840913 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.840882    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58tg6" event={"ID":"b059c4d3-70d1-4c04-95e9-e1735da565ef","Type":"ContainerStarted","Data":"6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524"}
Nov 04 01:03:26.842009 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.841966    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gfjzp" event={"ID":"4b6e932c-bb1d-424e-8dd2-c92797151de6","Type":"ContainerStarted","Data":"b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943"}
Nov 04 01:03:26.843102 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:26.843081    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7c8v" event={"ID":"bc5258b8-2791-49ac-a271-e914db291c6f","Type":"ContainerStarted","Data":"4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf"}
Nov 04 01:03:27.008162 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:27.008133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:27.266483 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:27.266450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:27.464569 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:27.464537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:28.212587 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:28.212546    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:28.854325 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:28.854286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:29.137594 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:29.137556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:29.358182 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:29.358150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:30.395121 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:30.395090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:30.595421 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:30.595385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:30.862313 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:30.861687    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58tg6" event={"ID":"b059c4d3-70d1-4c04-95e9-e1735da565ef","Type":"ContainerStarted","Data":"fec89d82cbfb53d661ee83835c2fd4054641a3668399e270a5449c6a03a147fa"}
Nov 04 01:03:30.862313 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:30.861828    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:30.880288 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:30.880234    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-58tg6" podStartSLOduration=67.405742329 podStartE2EDuration="1m10.880221055s" totalImagesPullingTime="3.474478726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:30.880098386 +0000 UTC m=+71.099802916" watchObservedRunningTime="2026-11-04 01:03:30.880221055 +0000 UTC m=+71.099925588"
Nov 04 01:03:32.287608 ip-10-0-140-186 systemd[1]: Stopping Kubernetes Kubelet...
Nov 04 01:03:32.288359 ip-10-0-140-186 kubenswrapper[2579]: I1104 01:03:32.288274    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 04 01:03:32.313208 ip-10-0-140-186 systemd[1]: kubelet.service: Deactivated successfully.
Nov 04 01:03:32.313359 ip-10-0-140-186 systemd[1]: Stopped Kubernetes Kubelet.
Nov 04 01:03:32.313592 ip-10-0-140-186 systemd[1]: kubelet.service: Consumed 4.580s CPU time.
Nov 04 01:03:32.317525 ip-10-0-140-186 systemd[1]: Starting Kubernetes Kubelet...
Nov 04 01:03:32.386387 ip-10-0-140-186 kubenswrapper[5797]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 04 01:03:32.386387 ip-10-0-140-186 kubenswrapper[5797]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 04 01:03:32.386387 ip-10-0-140-186 kubenswrapper[5797]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 04 01:03:32.386387 ip-10-0-140-186 kubenswrapper[5797]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 04 01:03:32.389360 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389280    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.389360 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389358    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.389360 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389361    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.389360 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389365    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389368    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389372    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389374    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389377    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389380    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389384    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389386    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389390    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389392    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389395    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389398    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389400    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389403    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389405    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389408    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389411    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389413    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389416    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389418    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.389540 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389421    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389423    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389426    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389429    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389431    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389434    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389437    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389440    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389443    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389449    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389452    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389455    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389457    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389460    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389463    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389465    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389468    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389470    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389473    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389475    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.390097 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389478    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389480    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389485    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389489    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389493    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389496    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389499    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389505    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389508    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389510    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389514    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389516    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389519    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389521    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389524    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389526    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389529    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389533    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389535    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.390687 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389538    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389541    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389544    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389547    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389549    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389552    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389555    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389558    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389561    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389563    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389566    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389568    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389571    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389573    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389576    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389578    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389581    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389583    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389587    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.391205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389591    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389595    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389598    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389601    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389603    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389606    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389609    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389611    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389614    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389616    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389619    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389622    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389624    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389627    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389630    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389633    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389635    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389638    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389641    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.391727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389643    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389647    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389650    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389652    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389655    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.389657    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390435    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390442    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390446    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390449    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390452    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390455    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390457    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390460    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390465    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390467    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390471    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390474    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390477    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390479    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.392221 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390482    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390485    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390487    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390490    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390493    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390495    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390498    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390501    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390504    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390506    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390509    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390511    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390514    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390517    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390520    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390522    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390525    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390527    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390530    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.392727 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390533    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390535    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390538    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390540    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390543    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390546    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390548    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390552    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390555    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390557    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390560    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390563    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390566    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390569    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390571    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390574    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390577    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390580    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390583    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.393225 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390585    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390588    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390590    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390593    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390596    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390598    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390601    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390605    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390608    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390611    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390614    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390617    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390619    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390622    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390624    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390627    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390629    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390632    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390635    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.393696 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390637    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390641    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390644    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390646    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390649    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390653    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390657    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390659    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390662    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390665    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390668    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390671    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390674    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390676    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390680    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390684    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390688    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390691    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390694    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390697    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.394189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390700    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390703    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390706    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390708    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390711    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390713    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390716    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390718    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390721    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390723    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390726    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390729    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390731    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390735    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.390739    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391247    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391254    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391260    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391264    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391267    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.394691 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391270    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391272    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391275    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391278    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391281    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391284    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391286    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391289    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391291    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391294    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391297    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391300    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391302    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391305    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391307    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391310    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391313    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391315    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391318    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391320    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.395222 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391323    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391326    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391328    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391330    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391334    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391339    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391342    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391345    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391348    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391351    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391354    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391356    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391359    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391362    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391364    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391367    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391370    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391373    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391375    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.395733 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391378    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391380    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391383    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391385    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391388    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391391    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391393    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391396    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391399    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391401    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391404    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391406    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391409    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391413    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391416    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391419    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391422    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391425    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391428    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.396237 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391431    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391434    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391437    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391439    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391442    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391446    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391448    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391451    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391454    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391457    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391460    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391462    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391465    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391467    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391470    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391472    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391475    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391478    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391480    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.396705 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391483    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391485    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391488    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391490    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391493    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391495    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391498    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391501    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391503    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391506    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391508    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391511    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391514    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391516    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391519    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391521    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391524    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391526    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391529    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391532    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.397224 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391535    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391537    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391542    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.391545    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391630    5797 flags.go:64] FLAG: --address="0.0.0.0"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391641    5797 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391647    5797 flags.go:64] FLAG: --anonymous-auth="true"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391651    5797 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391656    5797 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391659    5797 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391663    5797 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391667    5797 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391670    5797 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391673    5797 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391676    5797 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391680    5797 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391683    5797 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391686    5797 flags.go:64] FLAG: --cgroup-root=""
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391689    5797 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391692    5797 flags.go:64] FLAG: --client-ca-file=""
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391695    5797 flags.go:64] FLAG: --cloud-provider="external"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391698    5797 flags.go:64] FLAG: --cluster-dns="[]"
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391703    5797 flags.go:64] FLAG: --cluster-domain=""
Nov 04 01:03:32.397717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391706    5797 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391709    5797 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391712    5797 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391715    5797 flags.go:64] FLAG: --container-log-max-files="5"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391719    5797 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391722    5797 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391725    5797 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391729    5797 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391732    5797 flags.go:64] FLAG: --contention-profiling="false"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391736    5797 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391739    5797 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391745    5797 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391748    5797 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391756    5797 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391761    5797 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391764    5797 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391767    5797 flags.go:64] FLAG: --enable-load-reader="false"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391771    5797 flags.go:64] FLAG: --enable-server="true"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391774    5797 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391778    5797 flags.go:64] FLAG: --event-burst="100"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391781    5797 flags.go:64] FLAG: --event-qps="50"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391784    5797 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391787    5797 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391790    5797 flags.go:64] FLAG: --eviction-hard=""
Nov 04 01:03:32.398297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391794    5797 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391797    5797 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391800    5797 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391803    5797 flags.go:64] FLAG: --eviction-soft=""
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391806    5797 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391809    5797 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391812    5797 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391815    5797 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391818    5797 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391821    5797 flags.go:64] FLAG: --fail-swap-on="true"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391824    5797 flags.go:64] FLAG: --feature-gates=""
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391828    5797 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391831    5797 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391834    5797 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391838    5797 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391841    5797 flags.go:64] FLAG: --healthz-port="10248"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391844    5797 flags.go:64] FLAG: --help="false"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391847    5797 flags.go:64] FLAG: --hostname-override="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391850    5797 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391855    5797 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391858    5797 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391862    5797 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391866    5797 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 04 01:03:32.398907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391869    5797 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391873    5797 flags.go:64] FLAG: --image-service-endpoint=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391875    5797 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391878    5797 flags.go:64] FLAG: --kube-api-burst="100"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391881    5797 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391884    5797 flags.go:64] FLAG: --kube-api-qps="50"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391887    5797 flags.go:64] FLAG: --kube-reserved=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391890    5797 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391893    5797 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391896    5797 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391899    5797 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391902    5797 flags.go:64] FLAG: --lock-file=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391905    5797 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391908    5797 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391911    5797 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391917    5797 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391920    5797 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391923    5797 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391926    5797 flags.go:64] FLAG: --logging-format="text"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391929    5797 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391932    5797 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391935    5797 flags.go:64] FLAG: --manifest-url=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391938    5797 flags.go:64] FLAG: --manifest-url-header=""
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391942    5797 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391946    5797 flags.go:64] FLAG: --max-open-files="1000000"
Nov 04 01:03:32.399500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391949    5797 flags.go:64] FLAG: --max-pods="110"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391952    5797 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391955    5797 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391960    5797 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391963    5797 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391966    5797 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391971    5797 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391974    5797 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391981    5797 flags.go:64] FLAG: --node-status-max-images="50"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.391984    5797 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392004    5797 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392007    5797 flags.go:64] FLAG: --pod-cidr=""
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392010    5797 flags.go:64] FLAG: --pod-manifest-path=""
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392013    5797 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392016    5797 flags.go:64] FLAG: --pods-per-core="0"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392019    5797 flags.go:64] FLAG: --port="10250"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392022    5797 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392025    5797 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c3fb62457c124b40"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392029    5797 flags.go:64] FLAG: --qos-reserved=""
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392032    5797 flags.go:64] FLAG: --read-only-port="10255"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392035    5797 flags.go:64] FLAG: --register-node="true"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392038    5797 flags.go:64] FLAG: --register-with-taints=""
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392042    5797 flags.go:64] FLAG: --registry-burst="10"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392045    5797 flags.go:64] FLAG: --registry-qps="5"
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392048    5797 flags.go:64] FLAG: --reserved-cpus=""
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392050    5797 flags.go:64] FLAG: --reserved-memory=""
Nov 04 01:03:32.400120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392054    5797 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392057    5797 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392061    5797 flags.go:64] FLAG: --rotate-certificates="false"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392063    5797 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392066    5797 flags.go:64] FLAG: --runonce="false"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392069    5797 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392072    5797 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392075    5797 flags.go:64] FLAG: --seccomp-default="false"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392078    5797 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392082    5797 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392087    5797 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392090    5797 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392095    5797 flags.go:64] FLAG: --storage-driver-password="root"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392098    5797 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392101    5797 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392104    5797 flags.go:64] FLAG: --storage-driver-user="root"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392106    5797 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392110    5797 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392113    5797 flags.go:64] FLAG: --system-cgroups=""
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392115    5797 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392120    5797 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392123    5797 flags.go:64] FLAG: --tls-cert-file=""
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392126    5797 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392131    5797 flags.go:64] FLAG: --tls-min-version=""
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392134    5797 flags.go:64] FLAG: --tls-private-key-file=""
Nov 04 01:03:32.400752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392137    5797 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392140    5797 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392143    5797 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392146    5797 flags.go:64] FLAG: --v="2"
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392150    5797 flags.go:64] FLAG: --version="false"
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392159    5797 flags.go:64] FLAG: --vmodule=""
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392163    5797 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392166    5797 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392290    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392294    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392296    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392299    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392302    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392305    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392307    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392310    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392314    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392317    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392320    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392324    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392326    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392329    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.401375 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392332    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392334    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392337    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392340    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392342    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392345    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392347    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392350    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392353    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392355    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392358    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392360    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392363    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392366    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392368    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392370    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392373    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392376    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392378    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392381    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.402072 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392383    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392386    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392388    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392391    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392393    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392396    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392400    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392402    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392405    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392409    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392412    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392415    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392418    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392421    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392423    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392426    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392428    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392431    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392434    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.402903 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392436    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392439    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392443    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392447    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392449    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392452    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392455    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392458    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392460    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392463    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392466    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392468    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392471    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392474    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392476    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392479    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392482    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392484    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392487    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.403663 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392491    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392495    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392498    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392503    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392506    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392509    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392512    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392515    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392517    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392520    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392522    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392525    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392528    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392530    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392533    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392535    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392538    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392541    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392543    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.404408 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392546    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392549    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392551    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392554    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392556    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392559    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392562    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392564    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392567    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392570    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392572    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392575    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392577    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392581    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392584    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.392609    5797 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 04 01:03:32.404934 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392645    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392651    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392654    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392657    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392660    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392662    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392665    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392668    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392670    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392673    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392676    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392678    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392681    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392683    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392686    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392689    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392692    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392694    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392697    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.405351 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392699    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392702    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392705    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392707    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392710    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392712    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392715    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392717    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392720    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392722    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392726    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392729    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392732    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392735    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392741    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392744    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392747    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392750    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392752    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392755    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.405830 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392757    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392760    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392763    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392765    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392768    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392770    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392773    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392775    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392778    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392780    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392783    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392786    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392788    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392791    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392793    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392796    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392798    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392801    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392804    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392806    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.406462 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392809    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392812    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392815    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392817    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392820    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392822    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392826    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392829    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392831    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392835    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392837    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392840    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392843    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392845    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392848    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392850    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392853    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392856    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392858    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392861    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.407053 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392863    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392866    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392869    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392871    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392874    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392876    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392881    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392885    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392888    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392890    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392893    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392896    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392898    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392901    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392904    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392907    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392909    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392912    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392916    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.407585 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392919    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392921    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392924    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392927    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392929    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392935    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392939    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.392942    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.396439    5797 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.396454    5797 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.396472    5797 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.396477    5797 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396538    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396543    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396548    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396550    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396554    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396556    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396559    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.408142 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396562    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396565    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396567    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396570    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396573    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396576    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396579    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396581    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396584    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396586    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396589    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396592    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396594    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396597    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396600    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396602    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396606    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396609    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396611    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.408611 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396614    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396616    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396619    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396622    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396625    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396628    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396630    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396633    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396636    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396639    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396641    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396644    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396646    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396649    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396652    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396655    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396657    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396660    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396664    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396668    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.409189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396671    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396674    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396676    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396679    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396682    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396685    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396688    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396690    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396693    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396695    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396699    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396701    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396704    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396706    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396709    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396713    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396716    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396719    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396721    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.409680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396724    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396727    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396730    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396732    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396735    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396738    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396740    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396743    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396745    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396748    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396751    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396753    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396756    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396758    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396761    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396764    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396766    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396769    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396771    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396774    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.410176 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396777    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396780    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396784    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396787    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396795    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396798    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396800    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396803    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396806    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396809    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396811    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396814    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396816    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396820    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396822    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396824    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396827    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396830    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396832    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396835    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.410716 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.396837    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.396864    5797 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397070    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397076    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397079    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397082    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397085    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397088    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397090    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397093    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397095    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397098    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397100    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397103    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397105    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397108    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.411244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397110    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397118    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397121    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397124    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397126    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397129    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397131    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397134    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397137    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397140    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397142    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397145    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397147    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397150    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397153    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397155    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397157    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397160    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397163    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.411647 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397166    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397170    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397173    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397176    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397179    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397181    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397184    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397187    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397189    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397192    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397194    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397197    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397199    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397202    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397205    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397212    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397220    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397223    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397226    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397229    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.412266 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397231    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397234    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397237    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397240    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397242    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397245    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397248    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397250    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397253    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397256    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397258    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397261    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397263    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397266    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397269    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397271    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397274    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397276    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397279    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397281    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.413306 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397284    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397286    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397289    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397292    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397294    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397297    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397299    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397302    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397311    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397314    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397317    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397319    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397322    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397324    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397327    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397330    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397332    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397335    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397338    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.413795 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397340    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397344    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397348    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397351    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397354    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397357    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397360    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397363    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397366    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397368    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397371    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397373    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397376    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397378    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.397402    5797 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397444    5797 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 04 01:03:32.414285 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397448    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397451    5797 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397454    5797 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397457    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397460    5797 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397463    5797 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397472    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397475    5797 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397477    5797 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397480    5797 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397482    5797 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397485    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397488    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397494    5797 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397497    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397499    5797 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397502    5797 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397504    5797 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397507    5797 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 04 01:03:32.414679 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397511    5797 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397515    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397517    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397520    5797 feature_gate.go:386] unrecognized feature gate: Example
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397523    5797 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397526    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397528    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397530    5797 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397533    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397536    5797 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397538    5797 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397541    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397543    5797 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397546    5797 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397548    5797 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397551    5797 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397554    5797 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397556    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397559    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397565    5797 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 04 01:03:32.415171 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397572    5797 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397575    5797 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397578    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397580    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397583    5797 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397585    5797 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397589    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397592    5797 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397594    5797 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397597    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397599    5797 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397602    5797 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397605    5797 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397607    5797 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397609    5797 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397612    5797 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397615    5797 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397617    5797 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397620    5797 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 04 01:03:32.415664 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397623    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397625    5797 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397628    5797 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397630    5797 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397633    5797 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397635    5797 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397638    5797 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397641    5797 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397645    5797 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397648    5797 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397651    5797 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397653    5797 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397657    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397659    5797 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397668    5797 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397671    5797 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397673    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397676    5797 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397679    5797 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397682    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 04 01:03:32.416199 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397685    5797 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397688    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397691    5797 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397694    5797 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397696    5797 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397699    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397701    5797 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397704    5797 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397707    5797 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397709    5797 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397712    5797 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397714    5797 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397717    5797 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397720    5797 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397723    5797 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397726    5797 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397729    5797 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397735    5797 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397738    5797 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397741    5797 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 04 01:03:32.416692 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397743    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397746    5797 feature_gate.go:386] unrecognized feature gate: Example2
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397749    5797 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397751    5797 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397755    5797 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397758    5797 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:32.397760    5797 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.397812    5797 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.398795    5797 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.398865    5797 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.399060    5797 server.go:1014] "Starting client certificate rotation"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.399276    5797 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.399390    5797 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:57:20 +0000 UTC" deadline="2028-06-27 23:20:51.460908016 +0000 UTC"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.399456    5797 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14446h17m19.061455214s"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.399767    5797 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.401034    5797 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 04 01:03:32.417235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.403963    5797 log.go:25] "Validated CRI v1 runtime API"
Nov 04 01:03:32.417641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.405931    5797 log.go:25] "Validated CRI v1 image API"
Nov 04 01:03:32.417641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.406756    5797 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 04 01:03:32.417641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.409045    5797 fs.go:135] Filesystem UUIDs: map[12ac6bd3-ecce-4867-a439-8d2d7656d4c7:/dev/nvme0n1p4 76143f64-01f5-43ed-bb53-56d92a0799fd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 04 01:03:32.417770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.409060    5797 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b1a5a1908961c7fe7f9e225e3cc8a299b0cd67aba699be13d27f5e07c9f785e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b1a5a1908961c7fe7f9e225e3cc8a299b0cd67aba699be13d27f5e07c9f785e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf/userdata/shm major:0 minor:459 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6293a177e01cda80b75f7a92986e2a5d86aa9a598c15c0184264a7589b194f01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6293a177e01cda80b75f7a92986e2a5d86aa9a598c15c0184264a7589b194f01/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305/userdata/shm major:0 minor:437 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/config-reloader/2 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/prometheus/15 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~empty-dir/config-out major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~projected/kube-api-access-nzzxm:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~projected/kube-api-access-nzzxm major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~projected/tls-assets major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/config major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/web-config major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~projected/kube-api-access-wrlwf:{mountpoint:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~projected/kube-api-access-wrlwf major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22778bfc-2110-48e0-85dd-25e5ca26c161/volumes/kubernetes.io~projected/kube-api-access-ll26k:{mountpoint:/var/lib/kubelet/pods/22778bfc-2110-48e0-85dd-25e5ca26c161/volumes/kubernetes.io~projected/kube-api-access-ll26k major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e0d88d-b649-469b-905b-25cded0252f9/volumes/kubernetes.io~projected/kube-api-access-2jp9w:{mountpoint:/var/lib/kubelet/pods/33e0d88d-b649-469b-905b-25cded0252f9/volumes/kubernetes.io~projected/kube-api-access-2jp9w major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~projected/kube-api-access-2hg5c:{mountpoint:/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~projected/kube-api-access-2hg5c major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6e932c-bb1d-424e-8dd2-c92797151de6/volumes/kubernetes.io~projected/kube-api-access-zdm5n:{mountpoint:/var/lib/kubelet/pods/4b6e932c-bb1d-424e-8dd2-c92797151de6/volumes/kubernetes.io~projected/kube-api-access-zdm5n major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6e932c-bb1d-424e-8dd2-c92797151de6/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4b6e932c-bb1d-424e-8dd2-c92797151de6/volumes/kubernetes.io~secret/metrics-certs major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5083bdb1-d4a2-4162-9b49-e3191fa61637/volumes/kubernetes.io~projected/kube-api-access-xgrfz:{mountpoint:/var/lib/kubelet/pods/5083bdb1-d4a2-4162-9b49-e3191fa61637/volumes/kubernetes.io~projected/kube-api-access-xgrfz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/alertmanager/9 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/config-reloader/6 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~empty-dir/config-out major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~projected/kube-api-access-szvh2:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~projected/kube-api-access-szvh2 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~projected/tls-assets major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/config-volume major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/web-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~projected/kube-api-access-gmdkv:{mountpoint:/var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~projected/kube-api-access-gmdkv major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~projected/kube-api-access-85l72:{mountpoint:/var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~projected/kube-api-access-85l72 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~projected/kube-api-access-lz5r7:{mountpoint:/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~projected/kube-api-access-lz5r7 major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~projected/kube-api-access-5zsw8:{mountpoint:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~projected/kube-api-access-5zsw8 major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~projected/kube-api-access-cg6wc:{mountpoint:/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~projected/kube-api-access-cg6wc major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95384914-9263-45e9-9639-3566767d7559/volumes/kubernetes.io~projected/kube-api-access-bvg8v:{mountpoint:/var/lib/kubelet/pods/95384914-9263-45e9-9639-3566767d7559/volumes/kubernetes.io~projected/kube-api-access-bvg8v major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~projected/kube-api-access-v8qjz:{mountpoint:/var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~projected/kube-api-access-v8qjz major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volumes/kubernetes.io~projected/kube-api-access-td4tb:{mountpoint:/var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volumes/kubernetes.io~projected/kube-api-access-td4tb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b059c4d3-70d1-4c04-95e9-e1735da565ef/volumes/kubernetes.io~projected/kube-api-access-4shxk:{mountpoint:/var/lib/kubelet/pods/b059c4d3-70d1-4c04-95e9-e1735da565ef/volumes/kubernetes.io~projected/kube-api-access-4shxk major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1de55b1-797e-4e61-aeee-624aa4e20ca5/volumes/kubernetes.io~projected/kube-api-access-nxcmd:{mountpoint:/var/lib/kubelet/pods/b1de55b1-797e-4e61-aeee-624aa4e20ca5/volumes/kubernetes.io~projected/kube-api-access-nxcmd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1de55b1-797e-4e61-aeee-624aa4e20ca5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b1de55b1-797e-4e61-aeee-624aa4e20ca5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~projected/kube-api-access-7c8mm:{mountpoint:/var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~projected/kube-api-access-7c8mm major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc5258b8-2791-49ac-a271-e914db291c6f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bc5258b8-2791-49ac-a271-e914db291c6f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd730006-869f-4fb3-b626-912e7530f33c/volumes/kubernetes.io~projected/kube-api-access-czblg:{mountpoint:/var/lib/kubelet/pods/bd730006-869f-4fb3-b626-912e7530f33c/volumes/kubernetes.io~projected/kube-api-access-czblg major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf31dccb-516a-4a97-aa86-9277ec17154d/volumes/kubernetes.io~projected/kube-api-access-cprmj:{mountpoint:/var/lib/kubelet/pods/bf31dccb-516a-4a97-aa86-9277ec17154d/volumes/kubernetes.io~projected/kube-api-access-cprmj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf31dccb-516a-4a97-aa86-9277ec17154d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/bf31dccb-516a-4a97-aa86-9277ec17154d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8fac715-d046-4c8f-af92-66684a66e0ae/volumes/kubernetes.io~projected/kube-api-access-8q8ln:{mountpoint:/var/lib/kubelet/pods/e8fac715-d046-4c8f-af92-66684a66e0ae/volumes/kubernetes.io~projected/kube-api-access-8q8ln major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f132805a-066a-4d03-bcb3-90c1dc9dc73e/volumes/kubernetes.io~projected/kube-api-access-t9mwd:{mountpoint:/var/lib/kubelet/pods/f132805a-066a-4d03-bcb3-90c1dc9dc73e/volumes/kubernetes.io~projected/kube-api-access-t9mwd major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f132805a-066a-4d03-bcb3-90c1dc9dc73e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f132805a-066a-4d03-bcb3-90c1dc9dc73e/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f8a65bed28bd5ad3742f26c2f7abaf25511529eeb9b2be9e392210f6926071c7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/38808130ed3d3ebccea5c10f83b8523d180cf4bf0a98d5439342ac8fc1f4e9a8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9e9fa44747f445ed586ce9e7d3cbccfcfde626917fa8197cef4155d546e2548b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/7a3ea1fd0852076e3d481e9e77d884880d10dac96db00091e22916b69b44d19b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/19ff757bdec0d0a031a2c5178ce1abe5c25aea9568c4730af48965240975851d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/101d1f971f73c7d569e4a56adec65d9f2ce6ac3cb490a905e4ceda60ab8dd400/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bc854e9c3156e2247420079d038d67a374b22eca079b4ade70a32b0723ffdf11/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/631feb1d7b09c778c1ad76a8603579bbd7b2b5e496d29d63b87b417c703682f7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/87631b79d11d987e932877dee2a8c174dd57ab9c0260fc3d420903a3abe3e36d/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a3cf4d236ee4249e2632ca7d787a35a74aafca0288c2b7e5c8913438b25c7450/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/521bf0cc8f0bbc64e500485ac7cb7920433a9fea2de685b088bb75bcc79d90c9/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ed31867815e48f1055bb9b4926dc2b1182cec9dacd7ab3f0a4df77df063083f9/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ff85abaf85cd27e7c96e2500f9f4416933957918ae68fa7eea2cb0dfca9805ef/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6d4de21fe385ee537005a864c5f47d8cf5562ae06373709ecb766559adc62cbf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/abc69d605733a495ef6a5ec69e9b844f298d4256fdb34b6e0b8698a903e6340c/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a841f36a6963463c906d1ec3f4d47e9b904ae0c87a976dc45d73448d8496dc45/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2a5a0a2ff56f62b4eaef94e42d0520129b84675ba0b18a3351df355e54d93897/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7b123e16c3ef081e5a9753ed2989c4f7ee408e1a4669cb74090baaa798f958c0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5e76d41b1781bc00ff1b977a5c760879500b8a78274aae752b24bf195a5c5753/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/07fe0a1f131428f9c8b7cec998307a67b28a3024ee1f089d84c121fc10ac862b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/367afcf747fa127c232d5a48da66131bce7e6e1127e37c27c51b2fca9749c09a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/51af17e2bcfee40fe26a1eb6b4ebbbd8dafe77fd6a250ca8ecd45d7d13963c16/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/03c52ff807bb164a8436941e71b2372efe7984a5c0f1774dd8c70840aa124214/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/7cf1c06cd90ea7b850c027f5cc1d2a1a0d72a9ebd384a1055b12e6d2cc4bb199/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9933e53656b979766254f346ed32d97d05cb6328e0f0f4b76c62ef06727f88c7/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/72dd28b52ef4d8ff9d4c7dbf602c4db2a36f168046f0cf82aab6505471654e97/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/c61038dcaf7604b7b8bc973415095c5fd5d4c6f418a1a6e51226bc1b47aa3fa1/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/1cd65d6222cf2a2f228b0eeb5fc482d7b154616e229b436fc304a09bdd6d5ea9/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/85b1ed019f7fa1d945aaa9c06e130ca6149a8a65e8b4f426712edba6b708d829/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/af97afa97d0b66dac5bc6c7b0b818117f7885bc204289be5ae032d0af77f9215/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c972520930be744cfe427c9b7e62f3b0974a298d9b54f145d0ab502a481084be/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/368c27d56fe1d072d6a80a1ca2ca5696061606b84b5c557f0bf905c801a74c95/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/de0fe2ee2706bf5915ab44dcbd969d8579ab34776873213a203efb3562e633cb/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4cabdf59a5b21f04146e37714dc5ecdac418a441d96c59bcc40155c6f6a2c4c3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/744a2f4d723b537fbdedd74e7339148188c776ddaa2cefb287f39ad1333b8cdd/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/ef4ccd615931a8e970ba4784bfbda50602d3e4062a6787d3a826c633c1c75d4f/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/e5e0eca001df26036c6ce8021d914fdd2cc103ed4af17aeb8423f77e10fc06e5/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/6ec4d5b9d784ddf0bd2d6fe3b5f5bbccce5f4dcf15d9c6745fca4ba227880f4f/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/ac5f9f523fc152a7248e963e6043df0546e0b04a67311eaaea9a99b4ec495202/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/7c99962d146811ca43ed071e418fac09a56fbb27d846b2b56905c723e259e31f/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/754d816a2621b98734f0a0633cbb7da54c9f2007c1bfe6ad9a5b577ab9a4e183/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a90505cd6ff459369fd0aaed98c93e87012c211800d7d66f089d47b056569c34/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/514ba0f5453b85ef7547a8e22fc23146d970bada09b9bcc2791f3d65f8d70124/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/f7a7eaf0d469e60fd74750252a783f7d46b509f50ccd3e45ec05f62d5e73bd0a/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/b1f635e618399f63cb0699bc1fdbbbe68ecf5a328b7187ee57956dc4d689fae4/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/b652910e904f1b0b8bec8d6bf5967e830aa8c85b54d53077b3579650fac05f3c/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/16718ce101b9c26c0830810c5a35aca438d77162eb9398d4b20a731a7269ea68/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/1bf09b3296be54ca5255b1afaa77451fb5ef368500a53d1e4db671423c1f3a42/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/d75c15604b490fb04bf5bf4f61db62207d54d84a57965a40c935a9a3d6fbadcc/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/08529d34f45509144bd032e5d18f5be6bb498a7daebe057081c0a62ae47e3505/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/8f11859f943e4b801165539745c4366a37adecfd44f45bbe3cc0368a647f02eb/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/1223e77774d194571d3587a0db9c65148572311c99bd3169769baa5483c3b1a6/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/849246a948a485da733af5e4ccedb108027405cd32247f6e03a4f1d4fb4e0704/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/e711e17a335eab10a4488ae8734d72db6a0e5e07ebac488efbc052a1e6c88ab6/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/7d071ad2a28a5e93ffe564869e0fe5e3c987bd824dc5a89f38d03991c5dda409/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/f2dffdd23a54d02ef3aefdafb1ba8828af162936dcf5492253c9ded7b2386def/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/f03bc44a84f68d20061838f5a87309e0866d308272e899118d333b69ab88e6db/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/dbcb067d307229bcf31111c1c82399b6d75415dcaa67df3f7080003eae8ff81a/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/c458cfdd3cd15730e3d2b4184d699a63a5ed693c7f050d30cb301dcfa63c32d2/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/3dd2b5242363be5692d0e3b92bfaed0621a026d31294b79f03ddcb603ae4c9f0/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/9f9f4160411ea2c774c251880ff0072a8546c22ea2ba5a4bc07ab1251dde4ea4/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/2673c9273dfd2493d3324e32f13758eb5a326089e8e9e2968843c99d02129436/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/5c8ef5655ac0b6b14ef7d4180ebfe620a8f1c3bbac8add11688285a3271c9405/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/f03d67d1ff00f3567394819b553560a90344872a14f25ba12417bf1fcca8b7bf/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/669d78fe536799f67894bdf99fe62922cddac59e73ee41fcec2d3548c14d3b31/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/6fbeba1868b5a6b2d49b89924ac31d532004366a860906d6abaca2eefbad1742/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/92c9ccad9adbe31767c9466d8a94d4b46b288870e444978794df6369c4ae1427/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/f6a18b141f7f8bdcd7b330b96fe42642a4aa8fe6787735748cf33d42dc74ebee/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/b602b9c0c73c83fc9b0e689491b0471d10de3f7bebdc7603cb95b98c8e6d367a/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/a5c0be588a2adb129a9dd30f70f023fe2d32d4d07375a1b96734d986d231d651/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/13e54b51a6a64e86598fbdee13c984c39cf4f7c3d0f61c16bcdd43a4111df49d/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6646ae3e529a6a19d3fdf96e60eef307a1185af40970672e9fd5e0fbd0fc8f97/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/87a9d33ef8eb7e44657e8818660f437dc4681c887f04a05257252fd894a81f56/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c17c407a4da0b93fdad6f8239c02064d446321b6af761ba8e54f61c09de2f64e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/79ebd463e4f596c283513cb027f031ade0f9f09d62dbf627d004ced716a26579/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/a2f3bbdba38158abf8924e72873c44734a70baee9fc4b1f25186e1e4ff6343d5/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/64eec43892d3dd9169458d3fcdcbeac18b9c7d0c788b0032bf03db2b398fb897/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ae26cfe3298f50018624be0996602265f351907caa9395363d190007bd776760/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/807ea018e934c59eae2ddc044bb1b27b7ae4e0cd26987f223aa18bbe058f2d50/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5ac6da02d68115a4725293d311ddbbc70f17cd2de2e8c541401f55b429f43bba/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 04 01:03:32.418304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.417818    5797 manager.go:217] Machine: {Timestamp:2026-11-04 01:03:32.417330579 +0000 UTC m=+0.059236823 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096704 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210c5e45ff94d7c824ac407ce6b64c SystemUUID:ec210c5e-45ff-94d7-c824-ac407ce6b64c BootID:f4fc85fa-fc9f-4990-b232-2e021380871e Filesystems:[{Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~projected/kube-api-access-2hg5c DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:400 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea916bfe8ccf1b947c734da1f1cf6799b308c8496e9ec8d2cadee8d87ff0deef/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f132805a-066a-4d03-bcb3-90c1dc9dc73e/volumes/kubernetes.io~projected/kube-api-access-t9mwd DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f132805a-066a-4d03-bcb3-90c1dc9dc73e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~projected/kube-api-access-7c8mm DeviceMajor:0 DeviceMinor:253 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95384914-9263-45e9-9639-3566767d7559/volumes/kubernetes.io~projected/kube-api-access-bvg8v DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:401 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0be5199e3838a7e189359a01a092e54c7d8a56ac2c4bc69cd062a63cae4c8e0a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:392 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b1a5a1908961c7fe7f9e225e3cc8a299b0cd67aba699be13d27f5e07c9f785e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd55d181f5bae9dfbd13553f2edb4392ba8e7dbd3ef697c6e7a299e52946dbec/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd730006-869f-4fb3-b626-912e7530f33c/volumes/kubernetes.io~projected/kube-api-access-czblg DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/912c89946779a6455713875ef5cfcf7ea2d722e325950043287a0a7b7284bc49/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6293a177e01cda80b75f7a92986e2a5d86aa9a598c15c0184264a7589b194f01/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2578b4acbef3e2de3fe43a8d4f9f30e608b6ab82000fdc3ce7833453a5ab462/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e6eb3d00bcde935d4ebf855df8e842b4986e12ce818dcca2f700b1ddc6ab258/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:398 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~projected/kube-api-access-lz5r7 DeviceMajor:0 DeviceMinor:436 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/245c2ed3c884e5c0ee9d8fde8d99754687d0cc3850113d626a2180beb97b1e52/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:394 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1de55b1-797e-4e61-aeee-624aa4e20ca5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:396 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4765c3abf2cd78a76b3c7180c5d2675db3e5c7676c63ae10be42374e9f85c9cf/userdata/shm DeviceMajor:0 DeviceMinor:459 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:274 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:337 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:397 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5083bdb1-d4a2-4162-9b49-e3191fa61637/volumes/kubernetes.io~projected/kube-api-access-xgrfz DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/bf31dccb-516a-4a97-aa86-9277ec17154d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:298 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b059c4d3-70d1-4c04-95e9-e1735da565ef/volumes/kubernetes.io~projected/kube-api-access-4shxk DeviceMajor:0 DeviceMinor:462 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b74438b5ebc2953f0e3febded2cc5c01c862f7ed1671284ed05a28470101a943/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc5258b8-2791-49ac-a271-e914db291c6f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:455 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6e932c-bb1d-424e-8dd2-c92797151de6/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:461 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e68a36ee69b903ab8749c360b647c544576853b929ce712a0c34b560de90524/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfb2b1272d124d70fcd8fd81d0642b15996ac2e8dfe84c861011526fa41af1b3/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5df0493-dbfc-4398-94fa-2d5ec7676197/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/600018715a8b73fd0c208570aae2bd6cfa4627aad54278393bb950b390f81973/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7cfd0c13692f075078c13f6c36b28a5bd42cb0e00ae124a6df488626f8c6a73/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:395 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~projected/kube-api-access-nzzxm DeviceMajor:0 DeviceMinor:402 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:431 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/22778bfc-2110-48e0-85dd-25e5ca26c161/volumes/kubernetes.io~projected/kube-api-access-ll26k DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf31dccb-516a-4a97-aa86-9277ec17154d/volumes/kubernetes.io~projected/kube-api-access-cprmj DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~projected/kube-api-access-cg6wc DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33e0d88d-b649-469b-905b-25cded0252f9/volumes/kubernetes.io~projected/kube-api-access-2jp9w DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/513cdfcf-ac71-4d24-aca9-1c81edb6f803/volumes/kubernetes.io~projected/kube-api-access-gmdkv DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9693b4e7-8f19-4e24-aac8-4fb293180bae/volumes/kubernetes.io~projected/kube-api-access-v8qjz DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:388 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305/userdata/shm DeviceMajor:0 DeviceMinor:437 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/77b967b9-f2b0-4b23-9484-038568e3a0ea/volumes/kubernetes.io~projected/kube-api-access-5zsw8 DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:400 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08c47afb6c45b9b1d04ac6618102d72a7e3e509b6596960d8cad02fee6a532dd/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:399 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e38076f-6754-46f2-b054-f0aa1976319e/volumes/kubernetes.io~projected/kube-api-access-td4tb DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e8fac715-d046-4c8f-af92-66684a66e0ae/volumes/kubernetes.io~projected/kube-api-access-8q8ln DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e3d35b3b9dfd0f13c230a481b4cae3272e45f14e04a241b5db5597147e9a13d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:400 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6e932c-bb1d-424e-8dd2-c92797151de6/volumes/kubernetes.io~projected/kube-api-access-zdm5n DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b1de55b1-797e-4e61-aeee-624aa4e20ca5/volumes/kubernetes.io~projected/kube-api-access-nxcmd DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:393 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:400 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/649a3cc975a0beb5d102dae7959f9840f349c738a3fde341a98f4d1fac961e96/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:435 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89df4dbf2c418f28c3b48870e0e4a4d36e311145be6d65587468c91887dd2164/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2e54ab-acb8-4611-bf2d-534053133f77/volumes/kubernetes.io~projected/kube-api-access-wrlwf DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/552f596f-96f2-4d27-9b5a-1c87f8dc864d/volumes/kubernetes.io~projected/kube-api-access-85l72 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes/kubernetes.io~projected/kube-api-access-szvh2 DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0be5199e3838a7e MacAddress:a2:91:b7:4c:5d:56 Speed:10000 Mtu:8901} {Name:152a3125d07dcf7 MacAddress:fe:4e:d6:ea:fb:1c Speed:10000 Mtu:8901} {Name:245c2ed3c884e5c MacAddress:b6:f9:29:e0:10:9a Speed:10000 Mtu:8901} {Name:4765c3abf2cd78a MacAddress:36:85:51:9f:a5:97 Speed:10000 Mtu:8901} {Name:500b65de559cf45 MacAddress:ca:3f:6d:3f:39:66 Speed:10000 Mtu:8901} {Name:5e6eb3d00bcde93 MacAddress:4e:33:80:c7:1e:e0 Speed:10000 Mtu:8901} {Name:6293a177e01cda8 MacAddress:96:a2:41:6b:66:82 Speed:10000 Mtu:8901} {Name:649a3cc975a0beb MacAddress:8a:8f:7f:ba:55:27 Speed:10000 Mtu:8901} {Name:6e68a36ee69b903 MacAddress:66:10:0a:e6:00:e0 Speed:10000 Mtu:8901} {Name:89df4dbf2c418f2 MacAddress:92:42:a4:49:c7:52 Speed:10000 Mtu:8901} {Name:98d170e1ab9f462 MacAddress:0a:c2:01:7f:6e:8b Speed:10000 Mtu:8901} {Name:9a6604302f71c4c MacAddress:ba:45:16:95:d4:2d Speed:10000 Mtu:8901} {Name:b74438b5ebc2953 MacAddress:46:6d:f5:78:45:03 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:eb:dd:cf:c5 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:74:ea:34:b4:4f Speed:0 Mtu:8901} {Name:dfaf2e42d2438fc MacAddress:ce:86:7d:2c:92:c7 Speed:10000 Mtu:8901} {Name:e2578b4acbef3e2 MacAddress:2a:57:f3:b7:70:30 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:eb:dd:cf:c5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:a0:c2:9e:d1:71 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:e0:a9:50:c4:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 04 01:03:32.418595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.418308    5797 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 04 01:03:32.418595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.418353    5797 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 04 01:03:32.418792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.418767    5797 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 04 01:03:32.418954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.418793    5797 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-186.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 04 01:03:32.419064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.418957    5797 topology_manager.go:143] "Creating topology manager with none policy"
Nov 04 01:03:32.419064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.418970    5797 container_manager_linux.go:311] "Creating device plugin manager"
Nov 04 01:03:32.419064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419006    5797 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 04 01:03:32.419064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419034    5797 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 04 01:03:32.419241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419130    5797 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 04 01:03:32.419413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419400    5797 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 04 01:03:32.419543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419532    5797 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 04 01:03:32.419598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419577    5797 kubelet.go:493] "Attempting to sync node with API server"
Nov 04 01:03:32.419598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419591    5797 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 04 01:03:32.419692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419605    5797 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 04 01:03:32.419692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419654    5797 kubelet.go:405] "Adding apiserver pod source"
Nov 04 01:03:32.419692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.419667    5797 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 04 01:03:32.421961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.420921    5797 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 04 01:03:32.421961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.420960    5797 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 04 01:03:32.422212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.422030    5797 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 04 01:03:32.425225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425211    5797 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 04 01:03:32.425225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425225    5797 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 04 01:03:32.425336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425326    5797 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 04 01:03:32.425542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425531    5797 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 04 01:03:32.425575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425560    5797 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 04 01:03:32.425657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425648    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 04 01:03:32.425688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425661    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 04 01:03:32.425688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425668    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 04 01:03:32.425688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425673    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 04 01:03:32.425688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425679    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 04 01:03:32.425688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425685    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425692    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425697    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425704    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425710    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425726    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425736    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425761    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 04 01:03:32.425822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.425768    5797 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 04 01:03:32.429215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.429202    5797 server.go:1263] "Started kubelet"
Nov 04 01:03:32.429628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.429258    5797 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 04 01:03:32.429628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.429302    5797 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 04 01:03:32.429628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.429380    5797 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 04 01:03:32.429819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.429698    5797 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 04 01:03:32.429732 ip-10-0-140-186 systemd[1]: Started Kubernetes Kubelet.
Nov 04 01:03:32.430435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.430420    5797 server.go:317] "Adding debug handlers to kubelet server"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.436080    5797 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.436565    5797 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.436705    5797 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:57:22 +0000 UTC" deadline="2028-07-26 00:44:54.83335185 +0000 UTC"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.436726    5797 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15119h41m22.396628818s"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.436760    5797 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-186.us-east-2.compute.internal\" not found"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.436799    5797 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.436806    5797 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.437038    5797 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.438805    5797 factory.go:55] Registering systemd factory
Nov 04 01:03:32.439130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.438825    5797 factory.go:223] Registration of the systemd container factory successfully
Nov 04 01:03:32.440545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.439153    5797 factory.go:153] Registering CRI-O factory
Nov 04 01:03:32.440545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.439164    5797 factory.go:223] Registration of the crio container factory successfully
Nov 04 01:03:32.440545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.439216    5797 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 04 01:03:32.440545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.439244    5797 factory.go:103] Registering Raw factory
Nov 04 01:03:32.440545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.439257    5797 manager.go:1199] Started watching for new ooms in manager
Nov 04 01:03:32.440545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.439707    5797 manager.go:319] Starting recovery of all containers
Nov 04 01:03:32.440832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.440672    5797 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:03:32.445366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445309    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445371    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445387    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e38076f-6754-46f2-b054-f0aa1976319e" volumeName="kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-config" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445400    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1de55b1-797e-4e61-aeee-624aa4e20ca5" volumeName="kubernetes.io/empty-dir/b1de55b1-797e-4e61-aeee-624aa4e20ca5-tmp-dir" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445414    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445427    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd730006-869f-4fb3-b626-912e7530f33c" volumeName="kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445451    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445465    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513cdfcf-ac71-4d24-aca9-1c81edb6f803" volumeName="kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445481    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445494    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445507    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9693b4e7-8f19-4e24-aac8-4fb293180bae" volumeName="kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-tls" seLinuxMountContext=""
Nov 04 01:03:32.445532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445519    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd730006-869f-4fb3-b626-912e7530f33c" volumeName="kubernetes.io/projected/bd730006-869f-4fb3-b626-912e7530f33c-kube-api-access-czblg" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445534    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="552f596f-96f2-4d27-9b5a-1c87f8dc864d" volumeName="kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-tuned" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445561    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445577    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1de55b1-797e-4e61-aeee-624aa4e20ca5" volumeName="kubernetes.io/configmap/b1de55b1-797e-4e61-aeee-624aa4e20ca5-config-volume" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445590    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445602    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e38076f-6754-46f2-b054-f0aa1976319e" volumeName="kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-script-lib" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445615    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf31dccb-516a-4a97-aa86-9277ec17154d" volumeName="kubernetes.io/configmap/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445625    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf31dccb-516a-4a97-aa86-9277ec17154d" volumeName="kubernetes.io/projected/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-api-access-cprmj" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445634    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445641    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513cdfcf-ac71-4d24-aca9-1c81edb6f803" volumeName="kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445649    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445657    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9693b4e7-8f19-4e24-aac8-4fb293180bae" volumeName="kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-metrics-client-ca" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445668    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9693b4e7-8f19-4e24-aac8-4fb293180bae" volumeName="kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445680    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445692    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445707    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5df0493-dbfc-4398-94fa-2d5ec7676197" volumeName="kubernetes.io/configmap/b5df0493-dbfc-4398-94fa-2d5ec7676197-metrics-client-ca" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445718    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" volumeName="kubernetes.io/configmap/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-konnectivity-ca" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445736    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445749    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-client-tls" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445762    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e38076f-6754-46f2-b054-f0aa1976319e" volumeName="kubernetes.io/projected/9e38076f-6754-46f2-b054-f0aa1976319e-kube-api-access-td4tb" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445777    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445798    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22778bfc-2110-48e0-85dd-25e5ca26c161" volumeName="kubernetes.io/projected/22778bfc-2110-48e0-85dd-25e5ca26c161-kube-api-access-ll26k" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445811    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513cdfcf-ac71-4d24-aca9-1c81edb6f803" volumeName="kubernetes.io/empty-dir/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-textfile" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445824    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-client-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445836    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513cdfcf-ac71-4d24-aca9-1c81edb6f803" volumeName="kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-tls" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445849    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445862    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445877    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/empty-dir/1a2e54ab-acb8-4611-bf2d-534053133f77-audit-log" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445890    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445903    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445916    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513cdfcf-ac71-4d24-aca9-1c81edb6f803" volumeName="kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-metrics-client-ca" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445929    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f132805a-066a-4d03-bcb3-90c1dc9dc73e" volumeName="kubernetes.io/secret/f132805a-066a-4d03-bcb3-90c1dc9dc73e-cert" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445946    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445958    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445971    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.445983    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.446045    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp" seLinuxMountContext=""
Nov 04 01:03:32.446275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.446058    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.446368    5797 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.446953    5797 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447227    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447257    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447266    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447275    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e38076f-6754-46f2-b054-f0aa1976319e" volumeName="kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-env-overrides" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447284    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5083bdb1-d4a2-4162-9b49-e3191fa61637" volumeName="kubernetes.io/projected/5083bdb1-d4a2-4162-9b49-e3191fa61637-kube-api-access-xgrfz" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447292    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447317    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447345    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6e932c-bb1d-424e-8dd2-c92797151de6" volumeName="kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447358    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447370    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447381    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513cdfcf-ac71-4d24-aca9-1c81edb6f803" volumeName="kubernetes.io/projected/513cdfcf-ac71-4d24-aca9-1c81edb6f803-kube-api-access-gmdkv" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447393    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8fac715-d046-4c8f-af92-66684a66e0ae" volumeName="kubernetes.io/projected/e8fac715-d046-4c8f-af92-66684a66e0ae-kube-api-access-8q8ln" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447405    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e0d88d-b649-469b-905b-25cded0252f9" volumeName="kubernetes.io/projected/33e0d88d-b649-469b-905b-25cded0252f9-kube-api-access-2jp9w" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447414    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5df0493-dbfc-4398-94fa-2d5ec7676197" volumeName="kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447430    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447441    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447449    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-federate-client-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447456    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5df0493-dbfc-4398-94fa-2d5ec7676197" volumeName="kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447465    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" volumeName="kubernetes.io/secret/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-agent-certs" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447472    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447479    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447489    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95384914-9263-45e9-9639-3566767d7559" volumeName="kubernetes.io/projected/95384914-9263-45e9-9639-3566767d7559-kube-api-access-bvg8v" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447496    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b059c4d3-70d1-4c04-95e9-e1735da565ef" volumeName="kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447504    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd730006-869f-4fb3-b626-912e7530f33c" volumeName="kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447513    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/projected/1a2e54ab-acb8-4611-bf2d-534053133f77-kube-api-access-wrlwf" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447520    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447528    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447535    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-metrics-client-ca" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447542    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6e932c-bb1d-424e-8dd2-c92797151de6" volumeName="kubernetes.io/projected/4b6e932c-bb1d-424e-8dd2-c92797151de6-kube-api-access-zdm5n" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447549    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447558    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5df0493-dbfc-4398-94fa-2d5ec7676197" volumeName="kubernetes.io/projected/b5df0493-dbfc-4398-94fa-2d5ec7676197-kube-api-access-7c8mm" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447566    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf31dccb-516a-4a97-aa86-9277ec17154d" volumeName="kubernetes.io/secret/bf31dccb-516a-4a97-aa86-9277ec17154d-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447573    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447581    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447589    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447596    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95384914-9263-45e9-9639-3566767d7559" volumeName="kubernetes.io/empty-dir/95384914-9263-45e9-9639-3566767d7559-tmp-dir" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447604    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9693b4e7-8f19-4e24-aac8-4fb293180bae" volumeName="kubernetes.io/empty-dir/9693b4e7-8f19-4e24-aac8-4fb293180bae-volume-directive-shadow" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447611    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f132805a-066a-4d03-bcb3-90c1dc9dc73e" volumeName="kubernetes.io/projected/f132805a-066a-4d03-bcb3-90c1dc9dc73e-kube-api-access-t9mwd" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447618    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447628    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1b8f08-7323-48fd-9312-da6169e52c2a" volumeName="kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447636    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22778bfc-2110-48e0-85dd-25e5ca26c161" volumeName="kubernetes.io/configmap/22778bfc-2110-48e0-85dd-25e5ca26c161-serviceca" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447643    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447652    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="552f596f-96f2-4d27-9b5a-1c87f8dc864d" volumeName="kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-tmp" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447659    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e38076f-6754-46f2-b054-f0aa1976319e" volumeName="kubernetes.io/secret/9e38076f-6754-46f2-b054-f0aa1976319e-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447667    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447675    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc5258b8-2791-49ac-a271-e914db291c6f" volumeName="kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447683    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447690    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447698    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447706    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447715    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd730006-869f-4fb3-b626-912e7530f33c" volumeName="kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-binary-copy" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447723    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8fac715-d046-4c8f-af92-66684a66e0ae" volumeName="kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-daemon-config" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447731    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447739    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.447710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447747    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447756    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447763    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9693b4e7-8f19-4e24-aac8-4fb293180bae" volumeName="kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447771    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf31dccb-516a-4a97-aa86-9277ec17154d" volumeName="kubernetes.io/empty-dir/bf31dccb-516a-4a97-aa86-9277ec17154d-data-volume" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447778    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447785    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447795    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b967b9-f2b0-4b23-9484-038568e3a0ea" volumeName="kubernetes.io/projected/77b967b9-f2b0-4b23-9484-038568e3a0ea-kube-api-access-5zsw8" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447803    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd816f5-e80b-4a1b-9b57-feadd931954f" volumeName="kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447810    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8fac715-d046-4c8f-af92-66684a66e0ae" volumeName="kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-cni-binary-copy" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447828    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1de55b1-797e-4e61-aeee-624aa4e20ca5" volumeName="kubernetes.io/projected/b1de55b1-797e-4e61-aeee-624aa4e20ca5-kube-api-access-nxcmd" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447835    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e0d88d-b649-469b-905b-25cded0252f9" volumeName="kubernetes.io/configmap/33e0d88d-b649-469b-905b-25cded0252f9-iptables-alerter-script" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447842    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447850    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="472ca873-6ba4-403c-9a6c-2c330a7ac131" volumeName="kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447857    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="552f596f-96f2-4d27-9b5a-1c87f8dc864d" volumeName="kubernetes.io/projected/552f596f-96f2-4d27-9b5a-1c87f8dc864d-kube-api-access-85l72" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447864    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1de55b1-797e-4e61-aeee-624aa4e20ca5" volumeName="kubernetes.io/secret/b1de55b1-797e-4e61-aeee-624aa4e20ca5-metrics-tls" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447871    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" volumeName="kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447879    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2e54ab-acb8-4611-bf2d-534053133f77" volumeName="kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447886    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447895    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447903    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447911    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1430b29f-afb4-4814-ab0b-d31458805413" volumeName="kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447918    5797 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9693b4e7-8f19-4e24-aac8-4fb293180bae" volumeName="kubernetes.io/projected/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-api-access-v8qjz" seLinuxMountContext=""
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447925    5797 reconstruct.go:97] "Volume reconstruction finished"
Nov 04 01:03:32.450386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.447930    5797 reconciler.go:29] "Reconciler: start to sync state"
Nov 04 01:03:32.451545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.451091    5797 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 04 01:03:32.469011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.468920    5797 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 04 01:03:32.470194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.470172    5797 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 04 01:03:32.470297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.470205    5797 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 04 01:03:32.470297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.470225    5797 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 04 01:03:32.470297 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.470264    5797 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 04 01:03:32.474141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.474122    5797 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 04 01:03:32.478970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.478941    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="a59aff40f277c5ad13ac6a639828eb521384f3b8dbbc403bb35b6bc540403324" exitCode=0
Nov 04 01:03:32.493043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.493018    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49" exitCode=0
Nov 04 01:03:32.499280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.499257    5797 generic.go:356] "Generic (PLEG): container finished" podID="8fe1c320288f5f6b9ae9ef95fa346d10" containerID="c715bad5d1b418c381d382b5602488fe51f4f0f4494bee887e33f5077cc22ed6" exitCode=0
Nov 04 01:03:32.521943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.521913    5797 generic.go:356] "Generic (PLEG): container finished" podID="513cdfcf-ac71-4d24-aca9-1c81edb6f803" containerID="8dbdb0c7914e22c26d48bcb67f5397f4c97bd8bc60101e30f66bf04e40067c14" exitCode=0
Nov 04 01:03:32.535563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.535527    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="4c64db676abc87f50f6e524d63b72528c05866dbfa6df43f8b890f85f168ba13" exitCode=0
Nov 04 01:03:32.535684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.535568    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="2b8a6845621ae5374aa2374cde88f25b4b3796afd6cc03098a294260b8324258" exitCode=0
Nov 04 01:03:32.535749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.535669    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="553970ce697959ce39971d073d6908e7f9a3864805670c0cade12a1cd376c8de" exitCode=0
Nov 04 01:03:32.535749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.535705    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="4e140a0e6c87d5267b5bb6fa6ea795d3c1ce7f079fd8c96b65f008bee6b3e29e" exitCode=0
Nov 04 01:03:32.535749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.535718    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="37020ed7dbc942d5ec2f8efac53fdd08d980dcefe37367f4c0108470cb245133" exitCode=0
Nov 04 01:03:32.535749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.535729    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd730006-869f-4fb3-b626-912e7530f33c" containerID="547ccb0f6ae7cb104e9a48d9d061a9eeca6b03d8fd5c6259930c27ded49b38e2" exitCode=0
Nov 04 01:03:32.556936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.556919    5797 manager.go:324] Recovery completed
Nov 04 01:03:32.570354 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.570329    5797 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 04 01:03:32.582601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582582    5797 cpu_manager.go:226] "Starting" policy="none"
Nov 04 01:03:32.582601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582597    5797 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 04 01:03:32.582724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582614    5797 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 04 01:03:32.582756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582728    5797 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 04 01:03:32.582756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582738    5797 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 04 01:03:32.582756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582755    5797 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 04 01:03:32.582852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582761    5797 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 04 01:03:32.582852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582766    5797 policy_none.go:50] "Start"
Nov 04 01:03:32.582852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582809    5797 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 04 01:03:32.582852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582818    5797 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 04 01:03:32.582954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582946    5797 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 04 01:03:32.582954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582953    5797 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 04 01:03:32.583065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.582958    5797 policy_none.go:44] "Start"
Nov 04 01:03:32.587627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.587599    5797 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 04 01:03:32.590143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.590131    5797 manager.go:341] "Starting Device Plugin manager"
Nov 04 01:03:32.590603 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.590159    5797 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 04 01:03:32.590678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.590606    5797 server.go:87] "Starting device plugin registration server"
Nov 04 01:03:32.590905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.590892    5797 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 04 01:03:32.590966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.590909    5797 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 04 01:03:32.591041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.591023    5797 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 04 01:03:32.591178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.591128    5797 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 04 01:03:32.591234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.591180    5797 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 04 01:03:32.591234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.591188    5797 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 04 01:03:32.591331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.591258    5797 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:03:32.591167692Z","UUID":"8143c3e5-5b37-45be-9080-f9f5dfd626e0","Handler":null,"Name":"","Endpoint":""}
Nov 04 01:03:32.592761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.592743    5797 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 04 01:03:32.592858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.592786    5797 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 04 01:03:32.592904 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.592877    5797 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 04 01:03:32.691852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.691820    5797 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 04 01:03:32.694360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.694343    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:03:32.694438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.694376    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:03:32.694438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.694387    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:03:32.694438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.694436    5797 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.706217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.706190    5797 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.706272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.706263    5797 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.770886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.770780    5797 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"]
Nov 04 01:03:32.771546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771530    5797 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.771546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771523    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"8fe1c320288f5f6b9ae9ef95fa346d10","Type":"ContainerStarted","Data":"03933e1b7206041c38b654b7b575a01f023c8e4babc31ed46a7fc3d6f4401e2b"}
Nov 04 01:03:32.771640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771562    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"8fe1c320288f5f6b9ae9ef95fa346d10","Type":"ContainerDied","Data":"c715bad5d1b418c381d382b5602488fe51f4f0f4494bee887e33f5077cc22ed6"}
Nov 04 01:03:32.771679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771648    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"8fe1c320288f5f6b9ae9ef95fa346d10","Type":"ContainerStarted","Data":"8b3e77a0d75fa9ca615140b3910395aa510f1c8a2c55ee2667348a0b3728e04b"}
Nov 04 01:03:32.771679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771663    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"388b6cc67d09cb3e8a1219d1e84089fb","Type":"ContainerStarted","Data":"b2abef6f5151caa75e8437ad76069441f1239f318e4c54c0a0c9a768032813a2"}
Nov 04 01:03:32.771753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771679    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal" event={"ID":"388b6cc67d09cb3e8a1219d1e84089fb","Type":"ContainerStarted","Data":"1cd1b24e86174979495f82c54fe47b9b676b3ce1e5f6f3c0816e09d4a724faae"}
Nov 04 01:03:32.771753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.771747    5797 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.785522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.785500    5797 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 04 01:03:32.785610 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.785544    5797 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.785929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.785915    5797 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 04 01:03:32.785976 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:32.785948    5797 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.853330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.853297    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.853330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.853330    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.853498 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.853351    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/388b6cc67d09cb3e8a1219d1e84089fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"388b6cc67d09cb3e8a1219d1e84089fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.953870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.953838    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.953870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.953872    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.954073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.953912    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/388b6cc67d09cb3e8a1219d1e84089fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"388b6cc67d09cb3e8a1219d1e84089fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.954073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.953950    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/388b6cc67d09cb3e8a1219d1e84089fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"388b6cc67d09cb3e8a1219d1e84089fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.954073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.953954    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:32.954073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:32.953955    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fe1c320288f5f6b9ae9ef95fa346d10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" (UID: \"8fe1c320288f5f6b9ae9ef95fa346d10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:33.420235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.420188    5797 apiserver.go:52] "Watching apiserver"
Nov 04 01:03:33.439163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.439109    5797 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 04 01:03:33.441185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.441156    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-x7xvv","openshift-ovn-kubernetes/ovnkube-node-5fq7t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7","openshift-console/console-c845bd84f-szzk2","openshift-insights/insights-runtime-extractor-lbqr9","openshift-dns/node-resolver-6tjgh","openshift-multus/multus-additional-cni-plugins-fjpq6","openshift-cluster-node-tuning-operator/tuned-vf66l","openshift-monitoring/alertmanager-main-0","openshift-console/console-67f8d87f9b-d526x","openshift-multus/network-metrics-daemon-gfjzp","openshift-ingress-canary/ingress-canary-z5jfh","openshift-monitoring/node-exporter-29mzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal","openshift-console/console-6c94f7f5d6-x8mxr","openshift-monitoring/metrics-server-5bb86d5d94-r4w4m","openshift-monitoring/telemeter-client-b6c7df88-lkrmc","openshift-network-diagnostics/network-check-target-58tg6","openshift-multus/multus-jxdjh","openshift-dns/dns-default-z88mh","kube-system/global-pull-secret-syncer-x7c8v","openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn","openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk","kube-system/konnectivity-agent-6ch6l","kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal","openshift-image-registry/node-ca-dnjzj"]
Nov 04 01:03:33.446509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.446484    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 04 01:03:33.446623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.446484    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dh9vh\""
Nov 04 01:03:33.446623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.446584    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 04 01:03:33.446755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.446634    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n5n45\""
Nov 04 01:03:33.446755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.446697    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-k9wxw\""
Nov 04 01:03:33.447055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.447032    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 04 01:03:33.447055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.447043    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 04 01:03:33.447192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.447177    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.447544 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.447528    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 04 01:03:33.447639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.447564    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.447682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.447665    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.448226 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.448202    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h8pc2\""
Nov 04 01:03:33.448580 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.448560    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 04 01:03:33.448672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.448577    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 04 01:03:33.448672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.448616    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 04 01:03:33.448672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.448656    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.448832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.448736    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9c9zb\""
Nov 04 01:03:33.449578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.449550    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.449663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.449597    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.451359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.451017    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pvmvv\""
Nov 04 01:03:33.451359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.451127    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.451359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.451216    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 04 01:03:33.451706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.451637    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zw2l8\""
Nov 04 01:03:33.452265 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.452236    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78n7j\""
Nov 04 01:03:33.452368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.452330    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq94r\""
Nov 04 01:03:33.455001 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.454966    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 04 01:03:33.455210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455189    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.455389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455367    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 04 01:03:33.455562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455429    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 04 01:03:33.455562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455484    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.455562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455528    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.455562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455535    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.455811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455594    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.455811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455693    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 04 01:03:33.455811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455738    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.455811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455771    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.455811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455802    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5jp5j\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455835    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455889    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455908    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.455961    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456046    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456058    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.456089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456047    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.456387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456151    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 04 01:03:33.456387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456200    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.456528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456509    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.456971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456953    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.456971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.456961    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cfm9p\""
Nov 04 01:03:33.457134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457073    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 04 01:03:33.457134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457088    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 04 01:03:33.457216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457140    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 04 01:03:33.457216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457155    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.457216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457168    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 04 01:03:33.457400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457379    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vklqk\""
Nov 04 01:03:33.457462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457448    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 04 01:03:33.457502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.457449    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 04 01:03:33.458473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458456    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 04 01:03:33.458549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458458    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 04 01:03:33.458549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458484    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x6b8r\""
Nov 04 01:03:33.458784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458766    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 04 01:03:33.458849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458771    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 04 01:03:33.458849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458808    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458847    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458860    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458864    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7ffl4\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458872    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458876    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kzl6m\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458848    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458891    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhcr5\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458874    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6kj2d\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458911    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458926    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 04 01:03:33.458967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.458945    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 04 01:03:33.459608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.459085    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 04 01:03:33.460026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.459982    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 04 01:03:33.460026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460014    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 04 01:03:33.460026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460027    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 04 01:03:33.460236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460041    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 04 01:03:33.460525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460506    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gj86p\""
Nov 04 01:03:33.460525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460519    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 04 01:03:33.460525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460510    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 04 01:03:33.460708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460521    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 04 01:03:33.460708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460552    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 04 01:03:33.460708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460555    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 04 01:03:33.460708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460575    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 04 01:03:33.460708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460586    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6wds\""
Nov 04 01:03:33.460708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460651    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 04 01:03:33.460961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.460928    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 04 01:03:33.461667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.461638    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lkq85\""
Nov 04 01:03:33.461843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.461829    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-grc8c\""
Nov 04 01:03:33.462175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.462158    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4sne5ifg6e6mn\""
Nov 04 01:03:33.463804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.463787    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 04 01:03:33.540516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.540492    5797 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 04 01:03:33.542081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.542064    5797 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:33.542185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.542173    5797 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:33.545984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.545961    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 04 01:03:33.545984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.545970    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 04 01:03:33.545984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.545966    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546094    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546118    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546125    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546131    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546152    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546251    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546278    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546291    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-66v8h\""
Nov 04 01:03:33.546441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546323    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546693    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9426ur6dsotl2\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546697    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-48ts5\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546715    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546769    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546783    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546775    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546773    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 04 01:03:33.546876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.546825    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 04 01:03:33.556367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556340    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.556367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556366    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.556543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556386    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdm5n\" (UniqueName: \"kubernetes.io/projected/4b6e932c-bb1d-424e-8dd2-c92797151de6-kube-api-access-zdm5n\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:33.556543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556412    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf31dccb-516a-4a97-aa86-9277ec17154d-data-volume\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.556543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556489    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.556543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556496    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf31dccb-516a-4a97-aa86-9277ec17154d-data-volume\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.556543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556527    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-bin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556563    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-daemon-config\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556495    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556599    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-etc-kubernetes\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556627    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-modprobe-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556647    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556694    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556753    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.556843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556781    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-daemon-config\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556890    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556875    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556951    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556978    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.556923    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557019    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdkv\" (UniqueName: \"kubernetes.io/projected/513cdfcf-ac71-4d24-aca9-1c81edb6f803-kube-api-access-gmdkv\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557052    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-conf\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557057    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557079    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557093    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557104    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de55b1-797e-4e61-aeee-624aa4e20ca5-config-volume\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557127    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlwf\" (UniqueName: \"kubernetes.io/projected/1a2e54ab-acb8-4611-bf2d-534053133f77-kube-api-access-wrlwf\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557157    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.557178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557184    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.557743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557211    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.557743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557265    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.557743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557273    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.557743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557319    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de55b1-797e-4e61-aeee-624aa4e20ca5-config-volume\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.557743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557337    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.557388    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zsw8\" (UniqueName: \"kubernetes.io/projected/77b967b9-f2b0-4b23-9484-038568e3a0ea-kube-api-access-5zsw8\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558398    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1de55b1-797e-4e61-aeee-624aa4e20ca5-tmp-dir\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558444    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558459    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-netns\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558444    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558524    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558549    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558612    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8ln\" (UniqueName: \"kubernetes.io/projected/e8fac715-d046-4c8f-af92-66684a66e0ae-kube-api-access-8q8ln\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558647    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558650    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1de55b1-797e-4e61-aeee-624aa4e20ca5-tmp-dir\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558655    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558694    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558732    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-sys\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558770    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558805    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcmd\" (UniqueName: \"kubernetes.io/projected/b1de55b1-797e-4e61-aeee-624aa4e20ca5-kube-api-access-nxcmd\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558834    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-root\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558867    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-tmp\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558892    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558900    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558942    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.558976    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-ovn\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559038    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4tb\" (UniqueName: \"kubernetes.io/projected/9e38076f-6754-46f2-b054-f0aa1976319e-kube-api-access-td4tb\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559083    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8mm\" (UniqueName: \"kubernetes.io/projected/b5df0493-dbfc-4398-94fa-2d5ec7676197-kube-api-access-7c8mm\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559121    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-multus-certs\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559154    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg8v\" (UniqueName: \"kubernetes.io/projected/95384914-9263-45e9-9639-3566767d7559-kube-api-access-bvg8v\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559161    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559188    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559223    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-config\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559255    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-sys\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559302    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-metrics-client-ca\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559338    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559333    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-slash\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559475    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f132805a-066a-4d03-bcb3-90c1dc9dc73e-cert\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559506    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-device-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559539    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-cnibin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559572    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-federate-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559620    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559656    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559687    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559695    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559718    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czblg\" (UniqueName: \"kubernetes.io/projected/bd730006-869f-4fb3-b626-912e7530f33c-kube-api-access-czblg\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.559957    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560023    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560293    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560500    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560566    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-federate-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560729    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-tmp\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560806    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560840    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f132805a-066a-4d03-bcb3-90c1dc9dc73e-cert\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560885    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-metrics-client-ca\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560921    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560937    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5df0493-dbfc-4398-94fa-2d5ec7676197-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.560948    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561014    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-wtmp\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561064    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-metrics-client-ca\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561082    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-tls\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561116    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561153    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561198    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-metrics-client-ca\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561217    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561255    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561260    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22778bfc-2110-48e0-85dd-25e5ca26c161-host\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561298    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-serving-certs-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561341    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561354    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561494    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561537    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561635    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-config\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561644    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561713    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561750    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561788    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-systemd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561832    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-serving-certs-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561860    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-run\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561888    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.561898    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85l72\" (UniqueName: \"kubernetes.io/projected/552f596f-96f2-4d27-9b5a-1c87f8dc864d-kube-api-access-85l72\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562008    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562049    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-tls\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562052    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-system-cni-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562138    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562183    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-konnectivity-ca\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562266    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-script-lib\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562303    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprmj\" (UniqueName: \"kubernetes.io/projected/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-api-access-cprmj\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562251    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562373    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562401    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562431    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-k8s-cni-cncf-io\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562461    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-kubelet\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562430    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-konnectivity-ca\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562488    5797 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562515    5797 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:33.562572    5797 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562702    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562705    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-ovnkube-script-lib\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.562852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562902    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.562970    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563024    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:33.563043    5797 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563083    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-client-certs\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563117    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e38076f-6754-46f2-b054-f0aa1976319e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563143    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563174    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563304    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563461    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563611    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e38076f-6754-46f2-b054-f0aa1976319e-ovn-node-metrics-cert\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563637    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33e0d88d-b649-469b-905b-25cded0252f9-iptables-alerter-script\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563735    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-secret-metrics-server-client-certs\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563789    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563827    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563852    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-client-tls\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564263    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33e0d88d-b649-469b-905b-25cded0252f9-iptables-alerter-script\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564314    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.563797    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564455    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564477    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-netns\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564507    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564527    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564557    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-client-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564590    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-systemd-units\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564643    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564716    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564783    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564828    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-kubernetes\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564870    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jp9w\" (UniqueName: \"kubernetes.io/projected/33e0d88d-b649-469b-905b-25cded0252f9-kube-api-access-2jp9w\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564882    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a2e54ab-acb8-4611-bf2d-534053133f77-client-ca-bundle\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.564931    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9693b4e7-8f19-4e24-aac8-4fb293180bae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565036    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9693b4e7-8f19-4e24-aac8-4fb293180bae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565063    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565088    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565148    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565187    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-system-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565217    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-conf-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565254    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565289    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565352    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565363    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565365    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565402    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a2e54ab-acb8-4611-bf2d-534053133f77-audit-log\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565437    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-agent-certs\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565465    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-netd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565501    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-var-lib-kubelet\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565515    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565539    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565466    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a2e54ab-acb8-4611-bf2d-534053133f77-audit-log\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565564    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565596    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95384914-9263-45e9-9639-3566767d7559-hosts-file\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565629    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565632    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565689    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-node-log\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565703    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6eb6955-2fed-4ca2-9b74-e278bb5eb79c-agent-certs\") pod \"konnectivity-agent-6ch6l\" (UID: \"d6eb6955-2fed-4ca2-9b74-e278bb5eb79c\") " pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565724    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-log-socket\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565764    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-dbus\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565792    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565800    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565813    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-tuned\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565874    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-cni-binary-copy\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565896    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565921    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-kubelet\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.565977    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-tuned\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566014    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566036    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-cnibin\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566077    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566117    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-kubelet-config\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566166    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566196    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fac715-d046-4c8f-af92-66684a66e0ae-cni-binary-copy\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566214    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566266    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf31dccb-516a-4a97-aa86-9277ec17154d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566664    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566705    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-hostroot\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566741    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566769    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-host\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566784    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566814    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysconfig\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566823    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf31dccb-516a-4a97-aa86-9277ec17154d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566943    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf31dccb-516a-4a97-aa86-9277ec17154d-crio-socket\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.566977    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qjz\" (UniqueName: \"kubernetes.io/projected/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-api-access-v8qjz\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567047    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567074    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567102    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567130    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-accelerators-collector-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567198    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95384914-9263-45e9-9639-3566767d7559-tmp-dir\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567225    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567251    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-tls\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567279    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-sys-fs\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567311    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrfz\" (UniqueName: \"kubernetes.io/projected/5083bdb1-d4a2-4162-9b49-e3191fa61637-kube-api-access-xgrfz\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567331    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdkv\" (UniqueName: \"kubernetes.io/projected/513cdfcf-ac71-4d24-aca9-1c81edb6f803-kube-api-access-gmdkv\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.567339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567338    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5df0493-dbfc-4398-94fa-2d5ec7676197-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567499    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567500    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdm5n\" (UniqueName: \"kubernetes.io/projected/4b6e932c-bb1d-424e-8dd2-c92797151de6-kube-api-access-zdm5n\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567585    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5df0493-dbfc-4398-94fa-2d5ec7676197-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567622    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567659    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567695    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-var-lib-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567695    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567738    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-env-overrides\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567780    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-accelerators-collector-config\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567793    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95384914-9263-45e9-9639-3566767d7559-tmp-dir\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567819    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-lib-modules\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567854    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567900    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567908    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567928    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-socket-dir-parent\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567942    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e38076f-6754-46f2-b054-f0aa1976319e-env-overrides\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567942    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") pod \"console-6c94f7f5d6-x8mxr\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") " pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567957    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1de55b1-797e-4e61-aeee-624aa4e20ca5-metrics-tls\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.567966    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-tls\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568142    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mwd\" (UniqueName: \"kubernetes.io/projected/f132805a-066a-4d03-bcb3-90c1dc9dc73e-kube-api-access-t9mwd\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568165    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568173    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568147    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e932c-bb1d-424e-8dd2-c92797151de6-metrics-certs\") pod \"network-metrics-daemon-gfjzp\" (UID: \"4b6e932c-bb1d-424e-8dd2-c92797151de6\") " pod="openshift-multus/network-metrics-daemon-gfjzp"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568186    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1de55b1-797e-4e61-aeee-624aa4e20ca5-metrics-tls\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568203    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568230    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568277    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568312    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-multus\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568341    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22778bfc-2110-48e0-85dd-25e5ca26c161-serviceca\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568367    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll26k\" (UniqueName: \"kubernetes.io/projected/22778bfc-2110-48e0-85dd-25e5ca26c161-kube-api-access-ll26k\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568388    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9693b4e7-8f19-4e24-aac8-4fb293180bae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568393    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-socket-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568411    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568419    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568468    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-metrics-server-audit-profiles\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568501    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568530    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568655    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568698    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-bin\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568711    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568767    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568788    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a2e54ab-acb8-4611-bf2d-534053133f77-metrics-server-audit-profiles\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568795    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568809    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-textfile\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568857    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-textfile\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568906    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-etc-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568916    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22778bfc-2110-48e0-85dd-25e5ca26c161-serviceca\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.568977    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569034    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569076    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-registration-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569106    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-systemd\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569142    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569186    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569226    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569253    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569254    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569320    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569364    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569406    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569433    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33e0d88d-b649-469b-905b-25cded0252f9-host-slash\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569454    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77b967b9-f2b0-4b23-9484-038568e3a0ea-secret-telemeter-client\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569471    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569501    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569557    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd730006-869f-4fb3-b626-912e7530f33c-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569660    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569664    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569693    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569762    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569769    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc5258b8-2791-49ac-a271-e914db291c6f-original-pull-secret\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569953    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.569982    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.570657    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlwf\" (UniqueName: \"kubernetes.io/projected/1a2e54ab-acb8-4611-bf2d-534053133f77-kube-api-access-wrlwf\") pod \"metrics-server-5bb86d5d94-r4w4m\" (UID: \"1a2e54ab-acb8-4611-bf2d-534053133f77\") " pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571034    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcmd\" (UniqueName: \"kubernetes.io/projected/b1de55b1-797e-4e61-aeee-624aa4e20ca5-kube-api-access-nxcmd\") pod \"dns-default-z88mh\" (UID: \"b1de55b1-797e-4e61-aeee-624aa4e20ca5\") " pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571091    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571241    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.571246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571246    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8ln\" (UniqueName: \"kubernetes.io/projected/e8fac715-d046-4c8f-af92-66684a66e0ae-kube-api-access-8q8ln\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571369    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8mm\" (UniqueName: \"kubernetes.io/projected/b5df0493-dbfc-4398-94fa-2d5ec7676197-kube-api-access-7c8mm\") pod \"prometheus-operator-9cd89b74c-dxwgn\" (UID: \"b5df0493-dbfc-4398-94fa-2d5ec7676197\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dxwgn"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571618    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4tb\" (UniqueName: \"kubernetes.io/projected/9e38076f-6754-46f2-b054-f0aa1976319e-kube-api-access-td4tb\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571657    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zsw8\" (UniqueName: \"kubernetes.io/projected/77b967b9-f2b0-4b23-9484-038568e3a0ea-kube-api-access-5zsw8\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571972    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") pod \"console-67f8d87f9b-d526x\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") " pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.571983    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85l72\" (UniqueName: \"kubernetes.io/projected/552f596f-96f2-4d27-9b5a-1c87f8dc864d-kube-api-access-85l72\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.572183    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") pod \"alertmanager-main-0\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.572364    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg8v\" (UniqueName: \"kubernetes.io/projected/95384914-9263-45e9-9639-3566767d7559-kube-api-access-bvg8v\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.573528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.573516    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") pod \"console-c845bd84f-szzk2\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") " pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:33.574388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.574371    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b967b9-f2b0-4b23-9484-038568e3a0ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b6c7df88-lkrmc\" (UID: \"77b967b9-f2b0-4b23-9484-038568e3a0ea\") " pod="openshift-monitoring/telemeter-client-b6c7df88-lkrmc"
Nov 04 01:03:33.575553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.575537    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jp9w\" (UniqueName: \"kubernetes.io/projected/33e0d88d-b649-469b-905b-25cded0252f9-kube-api-access-2jp9w\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.575595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.575577    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czblg\" (UniqueName: \"kubernetes.io/projected/bd730006-869f-4fb3-b626-912e7530f33c-kube-api-access-czblg\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.577295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.577279    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprmj\" (UniqueName: \"kubernetes.io/projected/bf31dccb-516a-4a97-aa86-9277ec17154d-kube-api-access-cprmj\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.577557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.577542    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shxk\" (UniqueName: \"kubernetes.io/projected/b059c4d3-70d1-4c04-95e9-e1735da565ef-kube-api-access-4shxk\") pod \"network-check-target-58tg6\" (UID: \"b059c4d3-70d1-4c04-95e9-e1735da565ef\") " pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:33.578422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.578406    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrfz\" (UniqueName: \"kubernetes.io/projected/5083bdb1-d4a2-4162-9b49-e3191fa61637-kube-api-access-xgrfz\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.578533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.578512    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll26k\" (UniqueName: \"kubernetes.io/projected/22778bfc-2110-48e0-85dd-25e5ca26c161-kube-api-access-ll26k\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.578612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.578596    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mwd\" (UniqueName: \"kubernetes.io/projected/f132805a-066a-4d03-bcb3-90c1dc9dc73e-kube-api-access-t9mwd\") pod \"ingress-canary-z5jfh\" (UID: \"f132805a-066a-4d03-bcb3-90c1dc9dc73e\") " pod="openshift-ingress-canary/ingress-canary-z5jfh"
Nov 04 01:03:33.578711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.578696    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qjz\" (UniqueName: \"kubernetes.io/projected/9693b4e7-8f19-4e24-aac8-4fb293180bae-kube-api-access-v8qjz\") pod \"kube-state-metrics-5dc8d7d89b-bw7dk\" (UID: \"9693b4e7-8f19-4e24-aac8-4fb293180bae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bw7dk"
Nov 04 01:03:33.579681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.579660    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") pod \"prometheus-k8s-0\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:33.671084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671008    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-k8s-cni-cncf-io\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671040    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-k8s-cni-cncf-io\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671048    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-kubelet\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671077    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-kubelet\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671087    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-netns\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671125    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-systemd-units\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671140    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-kubernetes\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671158    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-system-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671172    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-conf-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671177    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-systemd-units\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671200    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-netd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671208    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-netns\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671222    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-var-lib-kubelet\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671237    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-netd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671235    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-kubernetes\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671248    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95384914-9263-45e9-9639-3566767d7559-hosts-file\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671248    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-conf-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671246    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-system-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671282    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-var-lib-kubelet\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671285    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-node-log\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671308    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-node-log\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671311    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-log-socket\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671289    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95384914-9263-45e9-9639-3566767d7559-hosts-file\") pod \"node-resolver-6tjgh\" (UID: \"95384914-9263-45e9-9639-3566767d7559\") " pod="openshift-dns/node-resolver-6tjgh"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671326    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-dbus\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671340    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.671353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671359    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-log-socket\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671363    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-kubelet\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671393    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-cnibin\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671411    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-dbus\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671417    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671443    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-kubelet-config\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671454    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-kubelet\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671464    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671466    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-cnibin\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671466    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671480    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671501    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc5258b8-2791-49ac-a271-e914db291c6f-kubelet-config\") pod \"global-pull-secret-syncer-x7c8v\" (UID: \"bc5258b8-2791-49ac-a271-e914db291c6f\") " pod="kube-system/global-pull-secret-syncer-x7c8v"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671508    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671506    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-hostroot\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671536    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-host\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671551    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysconfig\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671565    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-host\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671572    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf31dccb-516a-4a97-aa86-9277ec17154d-crio-socket\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671570    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-hostroot\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671588    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671603    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysconfig\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671607    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-sys-fs\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671628    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-var-lib-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671632    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf31dccb-516a-4a97-aa86-9277ec17154d-crio-socket\") pod \"insights-runtime-extractor-lbqr9\" (UID: \"bf31dccb-516a-4a97-aa86-9277ec17154d\") " pod="openshift-insights/insights-runtime-extractor-lbqr9"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671644    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-lib-modules\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671643    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-sys-fs\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671671    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-var-lib-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671678    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-socket-dir-parent\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671676    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-cni-dir\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671710    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671716    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-lib-modules\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671729    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-multus\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671745    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-socket-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671750    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671761    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671773    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-multus\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671781    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671786    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-socket-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671778    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-bin\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671820    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-etc-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671839    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-registration-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671853    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-systemd\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671870    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33e0d88d-b649-469b-905b-25cded0252f9-host-slash\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671889    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-bin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671744    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-multus-socket-dir-parent\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671891    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-etc-openvswitch\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671799    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-cni-bin\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671910    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-registration-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671923    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33e0d88d-b649-469b-905b-25cded0252f9-host-slash\") pod \"iptables-alerter-x7xvv\" (UID: \"33e0d88d-b649-469b-905b-25cded0252f9\") " pod="openshift-network-operator/iptables-alerter-x7xvv"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671924    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-systemd\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671943    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-var-lib-cni-bin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671949    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-etc-kubernetes\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.671979    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-modprobe-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672013    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-etc-kubernetes\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672030    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-conf\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672061    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672063    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-modprobe-d\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672109    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672140    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-netns\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672149    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-etc-sysctl-conf\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672137    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672165    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-sys\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672174    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.672151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672196    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-root\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672219    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-ovn\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672239    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-multus-certs\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672260    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-sys\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672270    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-ovn\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672199    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-sys\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672284    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-slash\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672196    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-run-netns\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672220    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-root\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672289    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-host-run-multus-certs\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672320    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-host-slash\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672324    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-sys\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672327    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-device-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672359    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5083bdb1-d4a2-4162-9b49-e3191fa61637-device-dir\") pod \"aws-ebs-csi-driver-node-zcsk7\" (UID: \"5083bdb1-d4a2-4162-9b49-e3191fa61637\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672384    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-cnibin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672410    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-wtmp\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672424    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fac715-d046-4c8f-af92-66684a66e0ae-cnibin\") pod \"multus-jxdjh\" (UID: \"e8fac715-d046-4c8f-af92-66684a66e0ae\") " pod="openshift-multus/multus-jxdjh"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672427    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22778bfc-2110-48e0-85dd-25e5ca26c161-host\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672444    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22778bfc-2110-48e0-85dd-25e5ca26c161-host\") pod \"node-ca-dnjzj\" (UID: \"22778bfc-2110-48e0-85dd-25e5ca26c161\") " pod="openshift-image-registry/node-ca-dnjzj"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672448    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-systemd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672471    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e38076f-6754-46f2-b054-f0aa1976319e-run-systemd\") pod \"ovnkube-node-5fq7t\" (UID: \"9e38076f-6754-46f2-b054-f0aa1976319e\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672472    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-run\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672495    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/552f596f-96f2-4d27-9b5a-1c87f8dc864d-run\") pod \"tuned-vf66l\" (UID: \"552f596f-96f2-4d27-9b5a-1c87f8dc864d\") " pod="openshift-cluster-node-tuning-operator/tuned-vf66l"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672500    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-system-cni-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672511    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513cdfcf-ac71-4d24-aca9-1c81edb6f803-node-exporter-wtmp\") pod \"node-exporter-29mzn\" (UID: \"513cdfcf-ac71-4d24-aca9-1c81edb6f803\") " pod="openshift-monitoring/node-exporter-29mzn"
Nov 04 01:03:33.673924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.672520    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd730006-869f-4fb3-b626-912e7530f33c-system-cni-dir\") pod \"multus-additional-cni-plugins-fjpq6\" (UID: \"bd730006-869f-4fb3-b626-912e7530f33c\") " pod="openshift-multus/multus-additional-cni-plugins-fjpq6"
Nov 04 01:03:33.741264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:33.741235    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f8d87f9b-d526x"]
Nov 04 01:03:34.547689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:34.547651    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gfjzp" event={"ID":"4b6e932c-bb1d-424e-8dd2-c92797151de6","Type":"ContainerStarted","Data":"715c5f009eca89e92262fd7f824419140200820222d0b4532b6f96c1fae21afc"}
Nov 04 01:03:34.550985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:34.550955    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"66e87fba5c9816266e2b9adc49d4b2836661674527909f377e045da88adea6c9"}
Nov 04 01:03:34.551115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:34.551033    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"6129b3732c7b96291970066053085eab9ebfb3d2ea5d4bd0145f4af20d3f5bf8"}
Nov 04 01:03:34.551115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:34.551072    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"7cba36cadb7041f9a99975cb9590494bef74f1a9cd22812f8b48dc1e8f3776f1"}
Nov 04 01:03:34.551115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:34.551098    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerStarted","Data":"6544718f6c196edaa4d0c87e1fa5f281c5fa229415a973f3cadfec9538e89b2f"}
Nov 04 01:03:34.808710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:34.808629    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:35.007071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:35.007037    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:36.275108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:36.275075    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z88mh"
Nov 04 01:03:36.413288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:36.413254    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:36.605820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:36.605743    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:37.435032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:37.435000    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:37.890677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:37.890641    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:37.890846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:37.890834    5797 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:03:37.905537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:37.905511    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:38.339847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:38.339761    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:03:40.728243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:40.728208    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:41.462847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:41.462810    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:41.849838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:41.849756    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:42.449480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:42.449397    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:43.322759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:43.322728    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:43.323160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:43.322955    5797 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:03:44.156119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:44.156082    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:03:48.237807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:48.237778    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t"
Nov 04 01:03:49.897811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:49.897779    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:03:50.468981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.468941    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:50.660579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.660535    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:50.733831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.733767    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:50.737506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.737483    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:03:50.859410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.859379    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8dc999f8-b5bn9"]
Nov 04 01:03:50.861635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.861618    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:50.880268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:50.880238    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8dc999f8-b5bn9"]
Nov 04 01:03:51.040337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040246    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-serving-cert\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040292    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-oauth-config\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040311    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-config\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040445    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-trusted-ca-bundle\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040462    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-tmp\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040490    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-service-ca\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040511    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-oauth-serving-cert\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.040816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.040558    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b2g\" (UniqueName: \"kubernetes.io/projected/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-kube-api-access-t8b2g\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.044818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.044791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:51.141670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141640    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-trusted-ca-bundle\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141674    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-tmp\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141710    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-service-ca\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141734    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-oauth-serving-cert\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141750    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b2g\" (UniqueName: \"kubernetes.io/projected/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-kube-api-access-t8b2g\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141788    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-serving-cert\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141820    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-oauth-config\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.141887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.141847    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-config\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.142225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.142164    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-tmp\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.142296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.142258    5797 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 04 01:03:51.142695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.142669    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-config\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.142824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.142693    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-trusted-ca-bundle\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.142824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.142713    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-service-ca\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.142824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.142726    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-oauth-serving-cert\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.144462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.144440    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-serving-cert\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.144462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.144459    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-console-oauth-config\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.153871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.153849    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b2g\" (UniqueName: \"kubernetes.io/projected/f51cfd3b-b1ee-4361-aef7-d80d21dac86a-kube-api-access-t8b2g\") pod \"console-6b8dc999f8-b5bn9\" (UID: \"f51cfd3b-b1ee-4361-aef7-d80d21dac86a\") " pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.172185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.172163    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:03:51.251356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.251322    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:51.446392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.446261    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8dc999f8-b5bn9"]
Nov 04 01:03:51.448923 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:03:51.448892    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51cfd3b_b1ee_4361_aef7_d80d21dac86a.slice/crio-17ffa577b52533186d0bfd718afb023b960f8818b92013249842a422783bcafd WatchSource:0}: Error finding container 17ffa577b52533186d0bfd718afb023b960f8818b92013249842a422783bcafd: Status 404 returned error can't find the container with id 17ffa577b52533186d0bfd718afb023b960f8818b92013249842a422783bcafd
Nov 04 01:03:51.462202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.462182    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:03:51.618177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.618142    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8dc999f8-b5bn9" event={"ID":"f51cfd3b-b1ee-4361-aef7-d80d21dac86a","Type":"ContainerStarted","Data":"222ef6ac943b62a32a382d0dc5b9737e86ce530d7c00355bdbd4df2bc2a32e60"}
Nov 04 01:03:51.618375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.618195    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8dc999f8-b5bn9" event={"ID":"f51cfd3b-b1ee-4361-aef7-d80d21dac86a","Type":"ContainerStarted","Data":"17ffa577b52533186d0bfd718afb023b960f8818b92013249842a422783bcafd"}
Nov 04 01:03:51.639575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:51.639523    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b8dc999f8-b5bn9" podStartSLOduration=1.639510282 podStartE2EDuration="1.639510282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:51.63895239 +0000 UTC m=+19.280858651" watchObservedRunningTime="2026-11-04 01:03:51.639510282 +0000 UTC m=+19.281416532"
Nov 04 01:03:54.628326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.628288    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 04 01:03:54.628740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.628670    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="alertmanager" containerID="cri-o://ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc" gracePeriod=120
Nov 04 01:03:54.628833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.628766    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-metric" containerID="cri-o://42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321" gracePeriod=120
Nov 04 01:03:54.628905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.628859    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy" containerID="cri-o://1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e" gracePeriod=120
Nov 04 01:03:54.628967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.628930    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-web" containerID="cri-o://5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe" gracePeriod=120
Nov 04 01:03:54.629042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.628999    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="config-reloader" containerID="cri-o://7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca" gracePeriod=120
Nov 04 01:03:54.629100 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:54.629083    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="prom-label-proxy" containerID="cri-o://1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4" gracePeriod=120
Nov 04 01:03:55.636738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636704    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4" exitCode=0
Nov 04 01:03:55.636738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636730    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321" exitCode=0
Nov 04 01:03:55.636738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636736    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e" exitCode=0
Nov 04 01:03:55.636738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636742    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca" exitCode=0
Nov 04 01:03:55.636738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636746    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc" exitCode=0
Nov 04 01:03:55.637268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636772    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4"}
Nov 04 01:03:55.637268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636836    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321"}
Nov 04 01:03:55.637268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636863    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e"}
Nov 04 01:03:55.637268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636886    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca"}
Nov 04 01:03:55.637268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.636910    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc"}
Nov 04 01:03:55.879323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.879298    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:55.883379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883359    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883393    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883425    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883452    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883473    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883507    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883555    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883589    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883634    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883668    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883706    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883740    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.883811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883779    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:03:55.883811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.883793    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") pod \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\" (UID: \"50f13cde-bfd4-4cd7-87f4-7238e4634bf6\") "
Nov 04 01:03:55.884096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.884039    5797 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-main-db\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.884383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.884357    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:55.884548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.884400    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:55.886477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.886429    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:03:55.886730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.886709    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.886865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.886812    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.887246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.887204    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.887584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.887384    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.887911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.887875    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "kube-api-access-szvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:55.888113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.888083    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:55.888201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.888181    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.891518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.891459    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.898393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.898314    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config" pod "50f13cde-bfd4-4cd7-87f4-7238e4634bf6" (UID: "50f13cde-bfd4-4cd7-87f4-7238e4634bf6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:55.984684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984650    5797 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-out\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984680    5797 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984692    5797 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984702    5797 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-main-tls\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984713    5797 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984721    5797 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-metrics-client-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984753    5797 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-web-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984764    5797 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984773    5797 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-tls-assets\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984782    5797 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-cluster-tls-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984791    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szvh2\" (UniqueName: \"kubernetes.io/projected/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-kube-api-access-szvh2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:55.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:55.984800    5797 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f13cde-bfd4-4cd7-87f4-7238e4634bf6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:56.250273 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.250190    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:03:56.643360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.643324    5797 generic.go:356] "Generic (PLEG): container finished" podID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerID="5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe" exitCode=0
Nov 04 01:03:56.643762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.643368    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe"}
Nov 04 01:03:56.643762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.643409    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 04 01:03:56.643762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.643426    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50f13cde-bfd4-4cd7-87f4-7238e4634bf6","Type":"ContainerDied","Data":"dfaf2e42d2438fca2b4d1fdf6d1dabd8e9ebd37e63c4f348a173c042c7b358bf"}
Nov 04 01:03:56.643762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.643458    5797 scope.go:122] "RemoveContainer" containerID="1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4"
Nov 04 01:03:56.651837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.651815    5797 scope.go:122] "RemoveContainer" containerID="42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321"
Nov 04 01:03:56.658728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.658712    5797 scope.go:122] "RemoveContainer" containerID="1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e"
Nov 04 01:03:56.664496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.664472    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 04 01:03:56.666308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.666293    5797 scope.go:122] "RemoveContainer" containerID="5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe"
Nov 04 01:03:56.669384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.669362    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 04 01:03:56.673771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.673754    5797 scope.go:122] "RemoveContainer" containerID="7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca"
Nov 04 01:03:56.680617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.680588    5797 scope.go:122] "RemoveContainer" containerID="ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc"
Nov 04 01:03:56.687486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.687466    5797 scope.go:122] "RemoveContainer" containerID="03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49"
Nov 04 01:03:56.694346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.694328    5797 scope.go:122] "RemoveContainer" containerID="1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4"
Nov 04 01:03:56.694624 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.694601    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4\": container with ID starting with 1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4 not found: ID does not exist" containerID="1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4"
Nov 04 01:03:56.694681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.694632    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4"} err="failed to get container status \"1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4\": rpc error: code = NotFound desc = could not find container \"1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4\": container with ID starting with 1701a5197e8be2df340f363247ba732207cb3a809b071729387ade4adb3285a4 not found: ID does not exist"
Nov 04 01:03:56.694681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.694666    5797 scope.go:122] "RemoveContainer" containerID="42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321"
Nov 04 01:03:56.694910 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.694889    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321\": container with ID starting with 42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321 not found: ID does not exist" containerID="42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321"
Nov 04 01:03:56.694981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.694920    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321"} err="failed to get container status \"42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321\": rpc error: code = NotFound desc = could not find container \"42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321\": container with ID starting with 42ed6f1b487456d0ae4e6a4a7ce14a9ed6f4e991f95a3c34bc63ed619756e321 not found: ID does not exist"
Nov 04 01:03:56.694981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.694943    5797 scope.go:122] "RemoveContainer" containerID="1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e"
Nov 04 01:03:56.695385 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.695365    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e\": container with ID starting with 1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e not found: ID does not exist" containerID="1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e"
Nov 04 01:03:56.695462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.695393    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e"} err="failed to get container status \"1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e\": rpc error: code = NotFound desc = could not find container \"1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e\": container with ID starting with 1e6e4ee98f5eb9d24ff99345a80d9d22f25fba2661fc46f262d000502954904e not found: ID does not exist"
Nov 04 01:03:56.695462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.695413    5797 scope.go:122] "RemoveContainer" containerID="5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe"
Nov 04 01:03:56.695648 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.695631    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe\": container with ID starting with 5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe not found: ID does not exist" containerID="5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe"
Nov 04 01:03:56.695701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.695653    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe"} err="failed to get container status \"5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe\": rpc error: code = NotFound desc = could not find container \"5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe\": container with ID starting with 5d8c561574c8e9a938dd7d44d6e6a586254ef00bccd86eaf15df59cdd5a399fe not found: ID does not exist"
Nov 04 01:03:56.695701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.695667    5797 scope.go:122] "RemoveContainer" containerID="7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca"
Nov 04 01:03:56.695908 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.695891    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca\": container with ID starting with 7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca not found: ID does not exist" containerID="7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca"
Nov 04 01:03:56.695970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.695917    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca"} err="failed to get container status \"7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca\": rpc error: code = NotFound desc = could not find container \"7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca\": container with ID starting with 7fa0903a869d272da4cf71019e825584d4bae66deb1a2a7f0b1bbccd07b3b5ca not found: ID does not exist"
Nov 04 01:03:56.695970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.695935    5797 scope.go:122] "RemoveContainer" containerID="ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc"
Nov 04 01:03:56.696237 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.696222    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc\": container with ID starting with ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc not found: ID does not exist" containerID="ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc"
Nov 04 01:03:56.696278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.696241    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc"} err="failed to get container status \"ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc\": rpc error: code = NotFound desc = could not find container \"ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc\": container with ID starting with ff10f112e5ad2c886b58d7ff63c97232b652b4a7995b742e83c2da9e7b3d46fc not found: ID does not exist"
Nov 04 01:03:56.696278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.696254    5797 scope.go:122] "RemoveContainer" containerID="03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49"
Nov 04 01:03:56.696491 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:03:56.696475    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49\": container with ID starting with 03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49 not found: ID does not exist" containerID="03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49"
Nov 04 01:03:56.696536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:56.696497    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49"} err="failed to get container status \"03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49\": rpc error: code = NotFound desc = could not find container \"03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49\": container with ID starting with 03edce0fd25c4b95c4b128f54601b55ec7459e0bdf2054ba5493915f1de07e49 not found: ID does not exist"
Nov 04 01:03:58.472072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.472031    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c94f7f5d6-x8mxr" podUID="7fd816f5-e80b-4a1b-9b57-feadd931954f" containerName="console" containerID="cri-o://48fb538b782a141a5419f32c9e8fe764ae86b73edfb23ccb9c7571afbf33202f" gracePeriod=15
Nov 04 01:03:58.476491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.476464    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" path="/var/lib/kubelet/pods/50f13cde-bfd4-4cd7-87f4-7238e4634bf6/volumes"
Nov 04 01:03:58.653159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.653126    5797 generic.go:356] "Generic (PLEG): container finished" podID="7fd816f5-e80b-4a1b-9b57-feadd931954f" containerID="48fb538b782a141a5419f32c9e8fe764ae86b73edfb23ccb9c7571afbf33202f" exitCode=0
Nov 04 01:03:58.653278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.653193    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c94f7f5d6-x8mxr" event={"ID":"7fd816f5-e80b-4a1b-9b57-feadd931954f","Type":"ContainerDied","Data":"48fb538b782a141a5419f32c9e8fe764ae86b73edfb23ccb9c7571afbf33202f"}
Nov 04 01:03:58.730032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.729955    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:58.809216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809186    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809244    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809265    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809298    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809318    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809336    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809394    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809429    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") pod \"7fd816f5-e80b-4a1b-9b57-feadd931954f\" (UID: \"7fd816f5-e80b-4a1b-9b57-feadd931954f\") "
Nov 04 01:03:58.809661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809611    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:03:58.809811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809681    5797 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fd816f5-e80b-4a1b-9b57-feadd931954f-tmp\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.809811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809711    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:58.809811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809770    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:58.809811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.809776    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:58.810300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.810270    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:58.811831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.811803    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:58.811937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.811858    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "kube-api-access-cg6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:03:58.811937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.811865    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config" pod "7fd816f5-e80b-4a1b-9b57-feadd931954f" (UID: "7fd816f5-e80b-4a1b-9b57-feadd931954f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.885021    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.885423    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="prometheus" containerID="cri-o://b483a4ae86d16ba0260df6d7468b828c6ea67dcfe7ee7f5609580a839a10715c" gracePeriod=600
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.885772    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-thanos" containerID="cri-o://66e87fba5c9816266e2b9adc49d4b2836661674527909f377e045da88adea6c9" gracePeriod=600
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.885878    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy" containerID="cri-o://6129b3732c7b96291970066053085eab9ebfb3d2ea5d4bd0145f4af20d3f5bf8" gracePeriod=600
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.885954    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-web" containerID="cri-o://7cba36cadb7041f9a99975cb9590494bef74f1a9cd22812f8b48dc1e8f3776f1" gracePeriod=600
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.886036    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="thanos-sidecar" containerID="cri-o://6544718f6c196edaa4d0c87e1fa5f281c5fa229415a973f3cadfec9538e89b2f" gracePeriod=600
Nov 04 01:03:58.886293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.886115    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="config-reloader" containerID="cri-o://3edef76f89e8de9ebd150ee4476f4f2869a08e4f708725e3e4c395dbc0951314" gracePeriod=600
Nov 04 01:03:58.910099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910076    5797 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-trusted-ca-bundle\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.910203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910103    5797 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-oauth-serving-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.910203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910114    5797 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.910203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910123    5797 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.910203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910132    5797 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-oauth-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.910203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910140    5797 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fd816f5-e80b-4a1b-9b57-feadd931954f-console-serving-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:58.910203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:58.910148    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6wc\" (UniqueName: \"kubernetes.io/projected/7fd816f5-e80b-4a1b-9b57-feadd931954f-kube-api-access-cg6wc\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:59.570330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.570292    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67f8d87f9b-d526x" podUID="472ca873-6ba4-403c-9a6c-2c330a7ac131" containerName="console" containerID="cri-o://d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c" gracePeriod=15
Nov 04 01:03:59.664195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664163    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="66e87fba5c9816266e2b9adc49d4b2836661674527909f377e045da88adea6c9" exitCode=0
Nov 04 01:03:59.664195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664189    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="6129b3732c7b96291970066053085eab9ebfb3d2ea5d4bd0145f4af20d3f5bf8" exitCode=0
Nov 04 01:03:59.664195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664197    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="7cba36cadb7041f9a99975cb9590494bef74f1a9cd22812f8b48dc1e8f3776f1" exitCode=0
Nov 04 01:03:59.664195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664206    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="6544718f6c196edaa4d0c87e1fa5f281c5fa229415a973f3cadfec9538e89b2f" exitCode=0
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664215    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="3edef76f89e8de9ebd150ee4476f4f2869a08e4f708725e3e4c395dbc0951314" exitCode=0
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664224    5797 generic.go:356] "Generic (PLEG): container finished" podID="1430b29f-afb4-4814-ab0b-d31458805413" containerID="b483a4ae86d16ba0260df6d7468b828c6ea67dcfe7ee7f5609580a839a10715c" exitCode=0
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664240    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"66e87fba5c9816266e2b9adc49d4b2836661674527909f377e045da88adea6c9"}
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664333    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"6129b3732c7b96291970066053085eab9ebfb3d2ea5d4bd0145f4af20d3f5bf8"}
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664365    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"7cba36cadb7041f9a99975cb9590494bef74f1a9cd22812f8b48dc1e8f3776f1"}
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664395    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"6544718f6c196edaa4d0c87e1fa5f281c5fa229415a973f3cadfec9538e89b2f"}
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664419    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"3edef76f89e8de9ebd150ee4476f4f2869a08e4f708725e3e4c395dbc0951314"}
Nov 04 01:03:59.664457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.664442    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"b483a4ae86d16ba0260df6d7468b828c6ea67dcfe7ee7f5609580a839a10715c"}
Nov 04 01:03:59.665504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.665485    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c94f7f5d6-x8mxr" event={"ID":"7fd816f5-e80b-4a1b-9b57-feadd931954f","Type":"ContainerDied","Data":"9a6604302f71c4cda8a22d3b82c62cd44c5b935ca121699fd4a2b3916e177212"}
Nov 04 01:03:59.665556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.665519    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c94f7f5d6-x8mxr"
Nov 04 01:03:59.665556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.665526    5797 scope.go:122] "RemoveContainer" containerID="48fb538b782a141a5419f32c9e8fe764ae86b73edfb23ccb9c7571afbf33202f"
Nov 04 01:03:59.685714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.685693    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c94f7f5d6-x8mxr"]
Nov 04 01:03:59.688117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.688099    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c94f7f5d6-x8mxr"]
Nov 04 01:03:59.813181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.813153    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:03:59.837854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.837830    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:03:59.917423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917394    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917478    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917497    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917535    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917657    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917703    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917731    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") pod \"472ca873-6ba4-403c-9a6c-2c330a7ac131\" (UID: \"472ca873-6ba4-403c-9a6c-2c330a7ac131\") "
Nov 04 01:03:59.917917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917806    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:03:59.918012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917864    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:59.918086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.917939    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:59.918086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.918068    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:03:59.918204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.918124    5797 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:59.918204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.918138    5797 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472ca873-6ba4-403c-9a6c-2c330a7ac131-tmp\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:59.918204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.918152    5797 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:59.918204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.918167    5797 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/472ca873-6ba4-403c-9a6c-2c330a7ac131-oauth-serving-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:03:59.919957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.919938    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:59.920200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.920180    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:03:59.920248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:03:59.920177    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c" pod "472ca873-6ba4-403c-9a6c-2c330a7ac131" (UID: "472ca873-6ba4-403c-9a6c-2c330a7ac131"). InnerVolumeSpecName "kube-api-access-2hg5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:04:00.019049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.019006    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hg5c\" (UniqueName: \"kubernetes.io/projected/472ca873-6ba4-403c-9a6c-2c330a7ac131-kube-api-access-2hg5c\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:00.019049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.019042    5797 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-serving-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:00.019049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.019054    5797 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/472ca873-6ba4-403c-9a6c-2c330a7ac131-console-oauth-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:00.475526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.475490    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd816f5-e80b-4a1b-9b57-feadd931954f" path="/var/lib/kubelet/pods/7fd816f5-e80b-4a1b-9b57-feadd931954f/volumes"
Nov 04 01:04:00.670948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.670910    5797 generic.go:356] "Generic (PLEG): container finished" podID="472ca873-6ba4-403c-9a6c-2c330a7ac131" containerID="d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c" exitCode=0
Nov 04 01:04:00.671330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.670956    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8d87f9b-d526x"
Nov 04 01:04:00.671330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.670978    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8d87f9b-d526x" event={"ID":"472ca873-6ba4-403c-9a6c-2c330a7ac131","Type":"ContainerDied","Data":"d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c"}
Nov 04 01:04:00.671330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.671037    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8d87f9b-d526x" event={"ID":"472ca873-6ba4-403c-9a6c-2c330a7ac131","Type":"ContainerDied","Data":"152a3125d07dcf7958855938eb56ae973a61a27477d59cb3ef3ec214477b54da"}
Nov 04 01:04:00.671330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.671060    5797 scope.go:122] "RemoveContainer" containerID="d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c"
Nov 04 01:04:00.678961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.678941    5797 scope.go:122] "RemoveContainer" containerID="d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c"
Nov 04 01:04:00.679269 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:04:00.679251    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c\": container with ID starting with d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c not found: ID does not exist" containerID="d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c"
Nov 04 01:04:00.679343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.679277    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c"} err="failed to get container status \"d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c\": rpc error: code = NotFound desc = could not find container \"d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c\": container with ID starting with d4beabd4a7335ba204c1e312d78e35e152c7587c73afc6784ee97ef9290e699c not found: ID does not exist"
Nov 04 01:04:00.689313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.689289    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f8d87f9b-d526x"]
Nov 04 01:04:00.693642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:00.693622    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f8d87f9b-d526x"]
Nov 04 01:04:01.172469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:01.172432    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:04:01.172631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:01.172586    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:04:01.176782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:01.176761    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:04:01.681466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:01.681432    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:04:01.730872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:01.730838    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c845bd84f-szzk2"]
Nov 04 01:04:02.478514 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:02.478478    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472ca873-6ba4-403c-9a6c-2c330a7ac131" path="/var/lib/kubelet/pods/472ca873-6ba4-403c-9a6c-2c330a7ac131/volumes"
Nov 04 01:04:03.142656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:03.142624    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:03.342131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:03.342103    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:06.412712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:06.412665    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:06.607962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:06.607923    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:08.342410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:08.342374    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-58tg6"
Nov 04 01:04:26.250172 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:04:26.250135    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:04:26.750195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:26.750129    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c845bd84f-szzk2" podUID="5e1b8f08-7323-48fd-9312-da6169e52c2a" containerName="console" containerID="cri-o://d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3" gracePeriod=15
Nov 04 01:04:26.989013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:26.988974    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:04:27.156932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.156898    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.156932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.156939    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157017    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157067    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157099    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157123    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157241    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157293    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") pod \"5e1b8f08-7323-48fd-9312-da6169e52c2a\" (UID: \"5e1b8f08-7323-48fd-9312-da6169e52c2a\") "
Nov 04 01:04:27.157399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157349    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:04:27.157556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157459    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:04:27.157556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157509    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:04:27.157665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157554    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:04:27.157665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157566    5797 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b8f08-7323-48fd-9312-da6169e52c2a-tmp\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.157665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157614    5797 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.157801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.157789    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:04:27.159484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.159456    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:04:27.159588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.159504    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "kube-api-access-lz5r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:04:27.159588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.159530    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert" pod "5e1b8f08-7323-48fd-9312-da6169e52c2a" (UID: "5e1b8f08-7323-48fd-9312-da6169e52c2a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:04:27.258209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.258173    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz5r7\" (UniqueName: \"kubernetes.io/projected/5e1b8f08-7323-48fd-9312-da6169e52c2a-kube-api-access-lz5r7\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.258209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.258201    5797 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-oauth-serving-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.258209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.258213    5797 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-oauth-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.258641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.258223    5797 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e1b8f08-7323-48fd-9312-da6169e52c2a-console-serving-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.258641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.258233    5797 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.258641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.258242    5797 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1b8f08-7323-48fd-9312-da6169e52c2a-trusted-ca-bundle\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:04:27.767718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.767684    5797 generic.go:356] "Generic (PLEG): container finished" podID="5e1b8f08-7323-48fd-9312-da6169e52c2a" containerID="d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3" exitCode=0
Nov 04 01:04:27.767890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.767724    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c845bd84f-szzk2" event={"ID":"5e1b8f08-7323-48fd-9312-da6169e52c2a","Type":"ContainerDied","Data":"d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3"}
Nov 04 01:04:27.767890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.767739    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c845bd84f-szzk2"
Nov 04 01:04:27.767890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.767760    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c845bd84f-szzk2" event={"ID":"5e1b8f08-7323-48fd-9312-da6169e52c2a","Type":"ContainerDied","Data":"98d170e1ab9f462d7bf9369cbcb02a68f1c8d591d5c596c12f2556ea33085305"}
Nov 04 01:04:27.767890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.767781    5797 scope.go:122] "RemoveContainer" containerID="d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3"
Nov 04 01:04:27.776081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.776064    5797 scope.go:122] "RemoveContainer" containerID="d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3"
Nov 04 01:04:27.776368 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:04:27.776347    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3\": container with ID starting with d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3 not found: ID does not exist" containerID="d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3"
Nov 04 01:04:27.776418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.776376    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3"} err="failed to get container status \"d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3\": rpc error: code = NotFound desc = could not find container \"d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3\": container with ID starting with d3e21e8309ea6e70b337a872c7a7f1e93a45e7c5d3044f935f8402170a64fec3 not found: ID does not exist"
Nov 04 01:04:27.790203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.790178    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c845bd84f-szzk2"]
Nov 04 01:04:27.795324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:27.795302    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c845bd84f-szzk2"]
Nov 04 01:04:28.478600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:28.478563    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1b8f08-7323-48fd-9312-da6169e52c2a" path="/var/lib/kubelet/pods/5e1b8f08-7323-48fd-9312-da6169e52c2a/volumes"
Nov 04 01:04:29.194519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:29.194484    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:53.657880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:53.657848    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:53.855592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:04:53.855561    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:04:56.249538 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:04:56.249501    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:05:10.762250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:05:10.762201    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:05:18.301511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:05:18.301429    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:05:18.499704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:05:18.499670    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:05:26.249387 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:05:26.249351    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:05:29.246651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:05:29.246615    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:05:52.359586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:05:52.359551    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:05:52.557198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:05:52.557164    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:05:56.249856 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:05:56.249814    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:05:59.012897 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:05:59.012854    5797 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL" podSandboxID="500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967"
Nov 04 01:05:59.013398 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:05:59.012905    5797 kuberuntime_manager.go:1939] "Failed to stop sandbox" podSandboxID={"Type":"cri-o","ID":"500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967"}
Nov 04 01:05:59.013398 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:05:59.012982    5797 kubelet.go:2243] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"1430b29f-afb4-4814-ab0b-d31458805413\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL\"" logger="UnhandledError"
Nov 04 01:05:59.013398 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:05:59.013020    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"1430b29f-afb4-4814-ab0b-d31458805413\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL\"" pod="openshift-monitoring/prometheus-k8s-0" podUID="1430b29f-afb4-4814-ab0b-d31458805413"
Nov 04 01:06:00.935420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:00.935394    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.014094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014067    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014101    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014126    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014185    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014226    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014260    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014292    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014321    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014348    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014389    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014423    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014451    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014477    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014505    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014532    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014559    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014603    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014522    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014632    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014643    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.014661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014665    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.015414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014696    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") pod \"1430b29f-afb4-4814-ab0b-d31458805413\" (UID: \"1430b29f-afb4-4814-ab0b-d31458805413\") "
Nov 04 01:06:01.015414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.014944    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.015414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.015009    5797 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.015414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.015031    5797 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-metrics-client-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.016154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.015799    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.016154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.015874    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:06:01.016563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.016533    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.017595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.017285    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.017686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.017668    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.018141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.017827    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:06:01.018141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.018078    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.018302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.018165    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "kube-api-access-nzzxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:06:01.018302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.018275    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:06:01.019809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.019767    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.019913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.019868    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.019975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.019946    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.020149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.020128    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.020224 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.020163    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:06:01.020262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.020239    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.020372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.020358    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.028855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.028754    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config" pod "1430b29f-afb4-4814-ab0b-d31458805413" (UID: "1430b29f-afb4-4814-ab0b-d31458805413"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:06:01.068418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.068389    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1430b29f-afb4-4814-ab0b-d31458805413","Type":"ContainerDied","Data":"500b65de559cf454886ff220b48ac2e5d380cdef18912f2b47dda259420b2967"}
Nov 04 01:06:01.068530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.068435    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.068530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.068482    5797 scope.go:122] "RemoveContainer" containerID="66e87fba5c9816266e2b9adc49d4b2836661674527909f377e045da88adea6c9"
Nov 04 01:06:01.075930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.075916    5797 scope.go:122] "RemoveContainer" containerID="6129b3732c7b96291970066053085eab9ebfb3d2ea5d4bd0145f4af20d3f5bf8"
Nov 04 01:06:01.082434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.082418    5797 scope.go:122] "RemoveContainer" containerID="7cba36cadb7041f9a99975cb9590494bef74f1a9cd22812f8b48dc1e8f3776f1"
Nov 04 01:06:01.088662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.088638    5797 scope.go:122] "RemoveContainer" containerID="6544718f6c196edaa4d0c87e1fa5f281c5fa229415a973f3cadfec9538e89b2f"
Nov 04 01:06:01.092050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.092029    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:06:01.095623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.095523    5797 scope.go:122] "RemoveContainer" containerID="3edef76f89e8de9ebd150ee4476f4f2869a08e4f708725e3e4c395dbc0951314"
Nov 04 01:06:01.096493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.096472    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:06:01.102359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.102339    5797 scope.go:122] "RemoveContainer" containerID="b483a4ae86d16ba0260df6d7468b828c6ea67dcfe7ee7f5609580a839a10715c"
Nov 04 01:06:01.109221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.109204    5797 scope.go:122] "RemoveContainer" containerID="a59aff40f277c5ad13ac6a639828eb521384f3b8dbbc403bb35b6bc540403324"
Nov 04 01:06:01.115385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115366    5797 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115387    5797 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-metrics-client-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115399    5797 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115409    5797 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115419    5797 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-web-config\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115428    5797 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115437    5797 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115446    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzxm\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-kube-api-access-nzzxm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115455    5797 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-kube-rbac-proxy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115464    5797 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-grpc-tls\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115472    5797 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115481    5797 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115495    5797 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-db\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115508    5797 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1430b29f-afb4-4814-ab0b-d31458805413-tls-assets\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115522    5797 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115535    5797 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1430b29f-afb4-4814-ab0b-d31458805413-config-out\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115544    5797 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1430b29f-afb4-4814-ab0b-d31458805413-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.115798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.115553    5797 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1430b29f-afb4-4814-ab0b-d31458805413-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:06:01.136672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136640    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:06:01.136896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136884    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="prom-label-proxy"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136897    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="prom-label-proxy"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136908    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="init-config-reloader"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136914    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="init-config-reloader"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136921    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="config-reloader"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136926    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="config-reloader"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136934    5797 cpu_manager.go:414] "containerMap: removing container" podUID="472ca873-6ba4-403c-9a6c-2c330a7ac131" containerName="console"
Nov 04 01:06:01.136941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136940    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472ca873-6ba4-403c-9a6c-2c330a7ac131" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136948    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="init-config-reloader"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136953    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="init-config-reloader"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136962    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="thanos-sidecar"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136966    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="thanos-sidecar"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136973    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="config-reloader"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136977    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="config-reloader"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.136985    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-web"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137006    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-web"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137014    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-thanos"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137018    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-thanos"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137027    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137031    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137037    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5e1b8f08-7323-48fd-9312-da6169e52c2a" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137041    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1b8f08-7323-48fd-9312-da6169e52c2a" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137047    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-metric"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137051    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-metric"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137059    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="alertmanager"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137063    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="alertmanager"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137070    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137074    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137082    5797 cpu_manager.go:414] "containerMap: removing container" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-web"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137086    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-web"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137091    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7fd816f5-e80b-4a1b-9b57-feadd931954f" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137095    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd816f5-e80b-4a1b-9b57-feadd931954f" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137101    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="prometheus"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137105    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="prometheus"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137144    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1b8f08-7323-48fd-9312-da6169e52c2a" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137150    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-metric"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137155    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="alertmanager"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137161    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137166    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="kube-rbac-proxy-web"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137170    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd816f5-e80b-4a1b-9b57-feadd931954f" containerName="console"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137175    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="prometheus"
Nov 04 01:06:01.137167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137181    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="prom-label-proxy"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137187    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="init-config-reloader"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137192    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f13cde-bfd4-4cd7-87f4-7238e4634bf6" containerName="config-reloader"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137196    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="472ca873-6ba4-403c-9a6c-2c330a7ac131" containerName="console"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137202    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="thanos-sidecar"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137208    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="config-reloader"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137213    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-web"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137218    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy-thanos"
Nov 04 01:06:01.138116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.137224    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430b29f-afb4-4814-ab0b-d31458805413" containerName="kube-rbac-proxy"
Nov 04 01:06:01.142031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.142011    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.147159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147140    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 04 01:06:01.147159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147153    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 04 01:06:01.147445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147427    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 04 01:06:01.147520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147445    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 04 01:06:01.147520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147500    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 04 01:06:01.147630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147558    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 04 01:06:01.147630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147575    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 04 01:06:01.147630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147613    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 04 01:06:01.147630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147624    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 04 01:06:01.147812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147614    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 04 01:06:01.147812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.147720    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 04 01:06:01.151105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.151069    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9426ur6dsotl2\""
Nov 04 01:06:01.151105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.151077    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-48ts5\""
Nov 04 01:06:01.151264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.151107    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 04 01:06:01.157094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.157063    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 04 01:06:01.157178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.157160    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 04 01:06:01.160430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.160406    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:06:01.216864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.216764    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.216864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.216805    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.216864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.216824    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.216960    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217013    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217061    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-config\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217089    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217113    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217130    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217153    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217217    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217259    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217278    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217293    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217321    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnc7\" (UniqueName: \"kubernetes.io/projected/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-kube-api-access-lsnc7\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217387    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217418    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217434    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-web-config\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217482    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.217525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.217529    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-config-out\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318311    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318351    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318370    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318386    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318409    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318432    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnc7\" (UniqueName: \"kubernetes.io/projected/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-kube-api-access-lsnc7\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318465    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318502    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318527    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-web-config\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318551    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.318610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318584    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-config-out\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318622    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318648    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318671    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318702    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318730    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318760    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-config\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318788    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318815    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.318843    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.319144    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.319238    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.319330    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.319756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.319635    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.320118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.320091    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.320217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.320163    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.320217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.320196    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.321524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.321494    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.321634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.321596    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.322579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.322299    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-config\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.322579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.322500    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.322975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.322804    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.322975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.322806    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.322975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.322860    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.323230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.323206    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.323787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.323765    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-web-config\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.323923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.323905    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.324399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.324379    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.324481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.324461    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-config-out\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.331248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.331226    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnc7\" (UniqueName: \"kubernetes.io/projected/32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd-kube-api-access-lsnc7\") pod \"prometheus-k8s-0\" (UID: \"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.471914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.471820    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:01.760740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:01.760711    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 04 01:06:01.762227 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:06:01.762196    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c1ed5e_0c1f_4cca_a6e6_fc16bf8ecefd.slice/crio-58ecdc7bb8fd644236fd8c9025e9c8e4ffbe4c9935f9d26a4538d001d883b79b WatchSource:0}: Error finding container 58ecdc7bb8fd644236fd8c9025e9c8e4ffbe4c9935f9d26a4538d001d883b79b: Status 404 returned error can't find the container with id 58ecdc7bb8fd644236fd8c9025e9c8e4ffbe4c9935f9d26a4538d001d883b79b
Nov 04 01:06:02.073315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:02.073231    5797 generic.go:356] "Generic (PLEG): container finished" podID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerID="45370e7cc2ced85373142e5dcbe0175d1c637c47d6f734b80d46ba69bae127bb" exitCode=0
Nov 04 01:06:02.073738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:02.073323    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerDied","Data":"45370e7cc2ced85373142e5dcbe0175d1c637c47d6f734b80d46ba69bae127bb"}
Nov 04 01:06:02.073738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:02.073419    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"58ecdc7bb8fd644236fd8c9025e9c8e4ffbe4c9935f9d26a4538d001d883b79b"}
Nov 04 01:06:02.475250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:02.475218    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1430b29f-afb4-4814-ab0b-d31458805413" path="/var/lib/kubelet/pods/1430b29f-afb4-4814-ab0b-d31458805413/volumes"
Nov 04 01:06:03.081045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.080985    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"178130aed03cdc7e96258e7f4e209d8f9a277751bf8fe7530cb1b8468c3257b1"}
Nov 04 01:06:03.081413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.081101    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"08fa4e069335d61bf651c2d5077821ae7b75da2d8e4fb2cbddb98049e4662ad5"}
Nov 04 01:06:03.081413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.081128    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"e539e3e35bdd83594e0c3a9ebf8a0e11e61dc579646a6e5018a0a6892e28044f"}
Nov 04 01:06:03.081413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.081152    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"3e3dbb9795d6c2c3836181e10b2d3d1f333320a4ee920d9aa1a8e8e9f223e51a"}
Nov 04 01:06:03.081413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.081176    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"b2aabbc8e69df1f68a9f51a9075cdff152a055985f39499428c5de73b3811e35"}
Nov 04 01:06:03.081413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.081198    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd","Type":"ContainerStarted","Data":"ea29130728343a1fec95a99f6dd813849ef0352a00c05133f33ba555ff882678"}
Nov 04 01:06:03.120368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:03.120294    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.104188531 podStartE2EDuration="2.12027725s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.088719ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:03.117686582 +0000 UTC m=+150.759592855" watchObservedRunningTime="2026-11-04 01:06:03.12027725 +0000 UTC m=+150.762183501"
Nov 04 01:06:06.478961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:06.478933    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:06:23.903287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:23.903251    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:24.116871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:24.116835    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:25.338357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:25.338318    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:25.537728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:25.537695    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:26.249251 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:06:26.249217    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:06:26.615833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:26.615805    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:29.296359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:29.296327    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:30.161295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.161260    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:30.371955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.371916    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:30.401387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.401339    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"]
Nov 04 01:06:30.405161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.405129    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.421417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.421385    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"]
Nov 04 01:06:30.425867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.425845    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 04 01:06:30.426044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.425882    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 04 01:06:30.426162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.426133    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-d6ng4\""
Nov 04 01:06:30.426162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.426073    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478388    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478441    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478474    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478499    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb2m\" (UniqueName: \"kubernetes.io/projected/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-kube-api-access-cxb2m\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478531    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478592    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478619    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478638    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.481010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.478675    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.568050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.567971    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:30.579586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579563    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb2m\" (UniqueName: \"kubernetes.io/projected/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-kube-api-access-cxb2m\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579597    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579634    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579653    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579668    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579691    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579733    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579759    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.579969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.579795    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.580139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.580108    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.580241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.580214    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.580417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.580395    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.580492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.580417    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.580492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.580441    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.582041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.582021    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.582343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.582321    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.591420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.591392    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.591746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.591725    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb2m\" (UniqueName: \"kubernetes.io/projected/0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b-kube-api-access-cxb2m\") pod \"openshift-ai-inference-openshift-ai-inference-597546666c-728zv\" (UID: \"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.725892 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.725806    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:30.761120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:30.761091    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:31.006692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:31.006619    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"]
Nov 04 01:06:31.009970 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:06:31.009941    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac6cf92_f241_4b0c_98e6_5ca2a56b9f9b.slice/crio-1f4caa13caf744c59981a66d8eb97f20350e892fe89ee9d35f0f2b56e8fca60a WatchSource:0}: Error finding container 1f4caa13caf744c59981a66d8eb97f20350e892fe89ee9d35f0f2b56e8fca60a: Status 404 returned error can't find the container with id 1f4caa13caf744c59981a66d8eb97f20350e892fe89ee9d35f0f2b56e8fca60a
Nov 04 01:06:31.011903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:31.011884    5797 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 01:06:31.177385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:31.177352    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" event={"ID":"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b","Type":"ContainerStarted","Data":"1f4caa13caf744c59981a66d8eb97f20350e892fe89ee9d35f0f2b56e8fca60a"}
Nov 04 01:06:33.372028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:33.371976    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:06:34.571941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:34.571913    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:06:34.572220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:34.571980    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:06:34.572220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:34.572063    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:06:35.192535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:35.192502    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" event={"ID":"0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b","Type":"ContainerStarted","Data":"7d6df4a13dfd489f0fb13deee5c600aa87228577b6d3fe083df979211003e9df"}
Nov 04 01:06:35.217206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:35.217103    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podStartSLOduration=1.6581957539999999 podStartE2EDuration="5.217086195s" totalImagesPullingTime="3.558890441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:35.21655698 +0000 UTC m=+182.858463267" watchObservedRunningTime="2026-11-04 01:06:35.217086195 +0000 UTC m=+182.858992445"
Nov 04 01:06:35.726676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:35.726646    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:35.730887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:35.730863    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:36.196420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:36.196389    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:36.197339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:06:36.197317    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:06:56.249422 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:06:56.249384    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:07:01.472889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:01.472842    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:07:01.488233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:01.488202    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:07:02.298923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:02.298895    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:07:05.597154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:05.597113    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:06.330765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.330724    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"]
Nov 04 01:07:06.341147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.341123    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.412512 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.412470    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"]
Nov 04 01:07:06.437086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.437044    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 04 01:07:06.437262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.437124    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 04 01:07:06.437262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.437171    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 04 01:07:06.437262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.437179    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ptvlp\""
Nov 04 01:07:06.437262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.437213    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 04 01:07:06.437421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.437384    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 04 01:07:06.504683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.504648    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3dba5003-a608-45f0-bee9-a0597e2d39b4-manifests\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.504683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.504686    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.504900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.504773    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkfrt\" (UniqueName: \"kubernetes.io/projected/3dba5003-a608-45f0-bee9-a0597e2d39b4-kube-api-access-mkfrt\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.504900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.504816    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.504900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.504865    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.505028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.504897    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dba5003-a608-45f0-bee9-a0597e2d39b4-tmp\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606111    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dba5003-a608-45f0-bee9-a0597e2d39b4-tmp\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606185    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3dba5003-a608-45f0-bee9-a0597e2d39b4-manifests\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606219    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606266    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkfrt\" (UniqueName: \"kubernetes.io/projected/3dba5003-a608-45f0-bee9-a0597e2d39b4-kube-api-access-mkfrt\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606321    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606369    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.606750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.606661    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3dba5003-a608-45f0-bee9-a0597e2d39b4-manifests\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.608414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.608390    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dba5003-a608-45f0-bee9-a0597e2d39b4-tmp\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.608676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.608652    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.608971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.608953    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.609056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.608970    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3dba5003-a608-45f0-bee9-a0597e2d39b4-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.618718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.618691    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkfrt\" (UniqueName: \"kubernetes.io/projected/3dba5003-a608-45f0-bee9-a0597e2d39b4-kube-api-access-mkfrt\") pod \"opendatahub-operator-controller-manager-8ff7fd4ff-hxltx\" (UID: \"3dba5003-a608-45f0-bee9-a0597e2d39b4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.657584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.657554    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:06.956770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:06.956733    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"]
Nov 04 01:07:06.961329 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:06.961301    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dba5003_a608_45f0_bee9_a0597e2d39b4.slice/crio-8f9e640b365be08cc2c3ab83a3758c968e2f3134dd54495a1bb5c3b9495e3b06 WatchSource:0}: Error finding container 8f9e640b365be08cc2c3ab83a3758c968e2f3134dd54495a1bb5c3b9495e3b06: Status 404 returned error can't find the container with id 8f9e640b365be08cc2c3ab83a3758c968e2f3134dd54495a1bb5c3b9495e3b06
Nov 04 01:07:07.303613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:07.303536    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" event={"ID":"3dba5003-a608-45f0-bee9-a0597e2d39b4","Type":"ContainerStarted","Data":"8f9e640b365be08cc2c3ab83a3758c968e2f3134dd54495a1bb5c3b9495e3b06"}
Nov 04 01:07:07.622176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:07.622139    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:09.205967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:09.205931    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:09.430704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:09.430671    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:09.643522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:09.643487    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:11.321071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.321034    5797 generic.go:356] "Generic (PLEG): container finished" podID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerID="d15b63ba1d8733ec6862ce64042c239a381745deaf7ec5e2dc7d44b476e89de1" exitCode=0
Nov 04 01:07:11.321456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.321120    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" event={"ID":"3dba5003-a608-45f0-bee9-a0597e2d39b4","Type":"ContainerDied","Data":"d15b63ba1d8733ec6862ce64042c239a381745deaf7ec5e2dc7d44b476e89de1"}
Nov 04 01:07:11.701708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.701666    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"]
Nov 04 01:07:11.704932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.704913    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:11.710356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.710334    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 04 01:07:11.710452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.710370    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 04 01:07:11.714503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.714482    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"]
Nov 04 01:07:11.727166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.727143    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-qf7lm\""
Nov 04 01:07:11.748881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.748858    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7-tmp\") pod \"jobset-operator-674cd59dc5-7nshw\" (UID: \"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:11.748983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.748889    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxz9l\" (UniqueName: \"kubernetes.io/projected/e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7-kube-api-access-vxz9l\") pod \"jobset-operator-674cd59dc5-7nshw\" (UID: \"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:11.849873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.849842    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7-tmp\") pod \"jobset-operator-674cd59dc5-7nshw\" (UID: \"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:11.849873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.849879    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxz9l\" (UniqueName: \"kubernetes.io/projected/e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7-kube-api-access-vxz9l\") pod \"jobset-operator-674cd59dc5-7nshw\" (UID: \"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:11.850290 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.850267    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7-tmp\") pod \"jobset-operator-674cd59dc5-7nshw\" (UID: \"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:11.862164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:11.862134    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxz9l\" (UniqueName: \"kubernetes.io/projected/e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7-kube-api-access-vxz9l\") pod \"jobset-operator-674cd59dc5-7nshw\" (UID: \"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:12.037695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:12.037613    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"
Nov 04 01:07:12.315627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:12.315527    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw"]
Nov 04 01:07:12.321009 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:12.320956    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode34d3f92_7bb9_4f14_8a2e_bd362ae0cfb7.slice/crio-de618c7a940d2a11d4356562907a82dd45e47f90facba012f94841e1f3e39607 WatchSource:0}: Error finding container de618c7a940d2a11d4356562907a82dd45e47f90facba012f94841e1f3e39607: Status 404 returned error can't find the container with id de618c7a940d2a11d4356562907a82dd45e47f90facba012f94841e1f3e39607
Nov 04 01:07:12.330067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:12.330033    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw" event={"ID":"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7","Type":"ContainerStarted","Data":"de618c7a940d2a11d4356562907a82dd45e47f90facba012f94841e1f3e39607"}
Nov 04 01:07:12.332436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:12.332415    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" event={"ID":"3dba5003-a608-45f0-bee9-a0597e2d39b4","Type":"ContainerStarted","Data":"6831f5a952f85c7bb575aafa033cb2b5e1083b9b70ba505e08ad913a35e216d4"}
Nov 04 01:07:12.332573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:12.332551    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:12.380456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:12.380400    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podStartSLOduration=2.210258326 podStartE2EDuration="6.380385724s" totalImagesPullingTime="3.757274842s" totalInitContainerRuntime="412.852556ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:12.378760495 +0000 UTC m=+220.020666769" watchObservedRunningTime="2026-11-04 01:07:12.380385724 +0000 UTC m=+220.022291970"
Nov 04 01:07:13.573918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:13.573875    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:16.349596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:16.349556    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw" event={"ID":"e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7","Type":"ContainerStarted","Data":"82eb4ed7055baf90ed4deb1edfa7956c689b3408fd9b5894b9ee1e18f2a3ca13"}
Nov 04 01:07:16.370692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:16.370645    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7nshw" podStartSLOduration=2.050959279 podStartE2EDuration="5.370632981s" totalImagesPullingTime="3.319673702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:16.368814746 +0000 UTC m=+224.010720997" watchObservedRunningTime="2026-11-04 01:07:16.370632981 +0000 UTC m=+224.012539231"
Nov 04 01:07:18.881179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:18.881142    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:19.985927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:19.985897    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:20.183437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:20.183403    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:21.845295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:21.845257    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:22.053833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:22.053798    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:22.557298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:22.557270    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:22.762433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:22.762399    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:23.183215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:23.183181    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:23.338488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:23.338461    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:07:23.695581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:23.695546    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:24.244185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:24.244154    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:25.020935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:25.020883    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:25.218876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:25.218837    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:26.249938 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:07:26.249905    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:07:27.385324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:27.385288    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:28.239172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:28.239140    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:29.191276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:29.191200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:29.405979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:29.405938    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:29.791142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:29.791113    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:29.992809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:29.992760    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:32.458608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:32.458550    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:33.037162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.037129    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:33.404974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.404943    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"]
Nov 04 01:07:33.409309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.409288    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.417272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.417247    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 04 01:07:33.418236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.418212    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"]
Nov 04 01:07:33.418354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.418302    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pp78l\""
Nov 04 01:07:33.418507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.418485    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 04 01:07:33.551711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.551673    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69fdf072-b1d9-4eca-83dd-fc22311d4a32-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ffngz\" (UID: \"69fdf072-b1d9-4eca-83dd-fc22311d4a32\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.552120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.551746    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22qj\" (UniqueName: \"kubernetes.io/projected/69fdf072-b1d9-4eca-83dd-fc22311d4a32-kube-api-access-p22qj\") pod \"cert-manager-webhook-5dbbbf75cc-ffngz\" (UID: \"69fdf072-b1d9-4eca-83dd-fc22311d4a32\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.653214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.653160    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22qj\" (UniqueName: \"kubernetes.io/projected/69fdf072-b1d9-4eca-83dd-fc22311d4a32-kube-api-access-p22qj\") pod \"cert-manager-webhook-5dbbbf75cc-ffngz\" (UID: \"69fdf072-b1d9-4eca-83dd-fc22311d4a32\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.653403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.653242    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69fdf072-b1d9-4eca-83dd-fc22311d4a32-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ffngz\" (UID: \"69fdf072-b1d9-4eca-83dd-fc22311d4a32\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.675303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.675237    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69fdf072-b1d9-4eca-83dd-fc22311d4a32-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ffngz\" (UID: \"69fdf072-b1d9-4eca-83dd-fc22311d4a32\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.682956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.682930    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22qj\" (UniqueName: \"kubernetes.io/projected/69fdf072-b1d9-4eca-83dd-fc22311d4a32-kube-api-access-p22qj\") pod \"cert-manager-webhook-5dbbbf75cc-ffngz\" (UID: \"69fdf072-b1d9-4eca-83dd-fc22311d4a32\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.721605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.721570    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:33.884269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:33.884235    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:34.002218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:34.002146    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"]
Nov 04 01:07:34.005234 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:34.005200    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69fdf072_b1d9_4eca_83dd_fc22311d4a32.slice/crio-4a66c31cc3e677558192688557b0295a0ec0913fcdb588dd6ced857aac12d33b WatchSource:0}: Error finding container 4a66c31cc3e677558192688557b0295a0ec0913fcdb588dd6ced857aac12d33b: Status 404 returned error can't find the container with id 4a66c31cc3e677558192688557b0295a0ec0913fcdb588dd6ced857aac12d33b
Nov 04 01:07:34.370313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:34.370280    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:34.422707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:34.422669    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" event={"ID":"69fdf072-b1d9-4eca-83dd-fc22311d4a32","Type":"ContainerStarted","Data":"4a66c31cc3e677558192688557b0295a0ec0913fcdb588dd6ced857aac12d33b"}
Nov 04 01:07:35.190386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:35.190353    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:36.814369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:36.814328    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:38.080371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.080334    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"]
Nov 04 01:07:38.083813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.083789    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:38.091602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.091575    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nxm2g\""
Nov 04 01:07:38.091685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.091600    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 04 01:07:38.091685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.091623    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 04 01:07:38.096641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.096618    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"]
Nov 04 01:07:38.200395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.200364    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmlml\" (UniqueName: \"kubernetes.io/projected/f493fdd3-b7e8-493f-af2e-df12f9651df0-kube-api-access-mmlml\") pod \"authorino-operator-7b7bb4bc77-7l6kd\" (UID: \"f493fdd3-b7e8-493f-af2e-df12f9651df0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:38.301297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.301257    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmlml\" (UniqueName: \"kubernetes.io/projected/f493fdd3-b7e8-493f-af2e-df12f9651df0-kube-api-access-mmlml\") pod \"authorino-operator-7b7bb4bc77-7l6kd\" (UID: \"f493fdd3-b7e8-493f-af2e-df12f9651df0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:38.316099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.316068    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmlml\" (UniqueName: \"kubernetes.io/projected/f493fdd3-b7e8-493f-af2e-df12f9651df0-kube-api-access-mmlml\") pod \"authorino-operator-7b7bb4bc77-7l6kd\" (UID: \"f493fdd3-b7e8-493f-af2e-df12f9651df0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:38.396596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.396569    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:38.439963 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.439931    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" event={"ID":"69fdf072-b1d9-4eca-83dd-fc22311d4a32","Type":"ContainerStarted","Data":"1b0bcf0780bfef3ce974eedb249d18a057312fa67d12dc6c83e4b701bf7b93fa"}
Nov 04 01:07:38.440144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.440046    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:38.464088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.463489    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podStartSLOduration=1.989151473 podStartE2EDuration="5.463469119s" totalImagesPullingTime="3.474317646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:38.462638735 +0000 UTC m=+246.104544994" watchObservedRunningTime="2026-11-04 01:07:38.463469119 +0000 UTC m=+246.105375369"
Nov 04 01:07:38.672333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:38.672309    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"]
Nov 04 01:07:38.674552 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:38.674524    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf493fdd3_b7e8_493f_af2e_df12f9651df0.slice/crio-5e4a4025eedc347d055a16029662609db89929caae92b3bb5aaeda7e9f17a922 WatchSource:0}: Error finding container 5e4a4025eedc347d055a16029662609db89929caae92b3bb5aaeda7e9f17a922: Status 404 returned error can't find the container with id 5e4a4025eedc347d055a16029662609db89929caae92b3bb5aaeda7e9f17a922
Nov 04 01:07:39.144045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.144011    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:39.357344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.357306    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c488b8c84-654z6"]
Nov 04 01:07:39.361929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.361906    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.370236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.370110    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 04 01:07:39.370236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.370179    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 04 01:07:39.370236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.370221    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j5w8w\""
Nov 04 01:07:39.378565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.378537    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c488b8c84-654z6"]
Nov 04 01:07:39.440286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.440205    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:39.444485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.444458    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" event={"ID":"f493fdd3-b7e8-493f-af2e-df12f9651df0","Type":"ContainerStarted","Data":"5e4a4025eedc347d055a16029662609db89929caae92b3bb5aaeda7e9f17a922"}
Nov 04 01:07:39.509640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.509603    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517f1a3c-9ae6-4bbb-a412-22101ed0784f-tls-certs\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.509903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.509881    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/517f1a3c-9ae6-4bbb-a412-22101ed0784f-tmp\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.509978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.509914    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxp2f\" (UniqueName: \"kubernetes.io/projected/517f1a3c-9ae6-4bbb-a412-22101ed0784f-kube-api-access-jxp2f\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.610536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.610501    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/517f1a3c-9ae6-4bbb-a412-22101ed0784f-tmp\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.610536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.610538    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxp2f\" (UniqueName: \"kubernetes.io/projected/517f1a3c-9ae6-4bbb-a412-22101ed0784f-kube-api-access-jxp2f\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.611055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.610559    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517f1a3c-9ae6-4bbb-a412-22101ed0784f-tls-certs\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.613094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.613065    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/517f1a3c-9ae6-4bbb-a412-22101ed0784f-tmp\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.613302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.613285    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517f1a3c-9ae6-4bbb-a412-22101ed0784f-tls-certs\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.622735 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.622710    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxp2f\" (UniqueName: \"kubernetes.io/projected/517f1a3c-9ae6-4bbb-a412-22101ed0784f-kube-api-access-jxp2f\") pod \"kube-auth-proxy-c488b8c84-654z6\" (UID: \"517f1a3c-9ae6-4bbb-a412-22101ed0784f\") " pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.664855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.664823    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:39.676757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.676734    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6"
Nov 04 01:07:39.886421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.886375    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:39.966831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:39.966790    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c488b8c84-654z6"]
Nov 04 01:07:39.973721 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:39.973684    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517f1a3c_9ae6_4bbb_a412_22101ed0784f.slice/crio-796e2fef6cfc462ab40c70f838e51ef308be6387298cfb291992f47b1c609d26 WatchSource:0}: Error finding container 796e2fef6cfc462ab40c70f838e51ef308be6387298cfb291992f47b1c609d26: Status 404 returned error can't find the container with id 796e2fef6cfc462ab40c70f838e51ef308be6387298cfb291992f47b1c609d26
Nov 04 01:07:40.451349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:40.451312    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6" event={"ID":"517f1a3c-9ae6-4bbb-a412-22101ed0784f","Type":"ContainerStarted","Data":"796e2fef6cfc462ab40c70f838e51ef308be6387298cfb291992f47b1c609d26"}
Nov 04 01:07:41.099263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.099226    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:41.321894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.321862    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:41.542018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.541973    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"]
Nov 04 01:07:41.562326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.562296    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:41.562891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.562757    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"]
Nov 04 01:07:41.594507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.594449    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:41.643114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.643070    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 04 01:07:41.643328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.643309    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kg255\""
Nov 04 01:07:41.733561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.733517    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmvj\" (UniqueName: \"kubernetes.io/projected/b96933e9-892e-41b6-9b0f-9568cb44c538-kube-api-access-gnmvj\") pod \"dns-operator-controller-manager-8557cbdb84-smsgb\" (UID: \"b96933e9-892e-41b6-9b0f-9568cb44c538\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:41.808045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.807946    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:41.834960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.834930    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmvj\" (UniqueName: \"kubernetes.io/projected/b96933e9-892e-41b6-9b0f-9568cb44c538-kube-api-access-gnmvj\") pod \"dns-operator-controller-manager-8557cbdb84-smsgb\" (UID: \"b96933e9-892e-41b6-9b0f-9568cb44c538\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:41.847937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.847900    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmvj\" (UniqueName: \"kubernetes.io/projected/b96933e9-892e-41b6-9b0f-9568cb44c538-kube-api-access-gnmvj\") pod \"dns-operator-controller-manager-8557cbdb84-smsgb\" (UID: \"b96933e9-892e-41b6-9b0f-9568cb44c538\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:41.879621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:41.879598    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:42.163564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.162112    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"]
Nov 04 01:07:42.165209 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:42.165182    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96933e9_892e_41b6_9b0f_9568cb44c538.slice/crio-1a3e4c616727b0f92afde7cb6d69de081001c2aac4cf9c8580c5de6d0d4ee0fb WatchSource:0}: Error finding container 1a3e4c616727b0f92afde7cb6d69de081001c2aac4cf9c8580c5de6d0d4ee0fb: Status 404 returned error can't find the container with id 1a3e4c616727b0f92afde7cb6d69de081001c2aac4cf9c8580c5de6d0d4ee0fb
Nov 04 01:07:42.324534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.324075    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:42.332208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.332182    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.338285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.338219    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:42.342178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.342142    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-czjxd\""
Nov 04 01:07:42.342274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.342241    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 04 01:07:42.424146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.424065    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:42.442223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.442189    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6l2\" (UniqueName: \"kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2\") pod \"limitador-limitador-74dfcfb894-l5k98\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.442383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.442306    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file\") pod \"limitador-limitador-74dfcfb894-l5k98\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.448347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.448318    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:42.463579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.463545    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" event={"ID":"b96933e9-892e-41b6-9b0f-9568cb44c538","Type":"ContainerStarted","Data":"1a3e4c616727b0f92afde7cb6d69de081001c2aac4cf9c8580c5de6d0d4ee0fb"}
Nov 04 01:07:42.465303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.465280    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" event={"ID":"f493fdd3-b7e8-493f-af2e-df12f9651df0","Type":"ContainerStarted","Data":"ab64906e6f1e25c98a3865b197ac88f9dd8cfd2b5277f474cabdbf6306b12b6a"}
Nov 04 01:07:42.465548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.465526    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:42.490396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.488242    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podStartSLOduration=1.339836424 podStartE2EDuration="4.488222683s" totalImagesPullingTime="3.148386259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:42.486518087 +0000 UTC m=+250.128424352" watchObservedRunningTime="2026-11-04 01:07:42.488222683 +0000 UTC m=+250.130128934"
Nov 04 01:07:42.543076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.543044    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6l2\" (UniqueName: \"kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2\") pod \"limitador-limitador-74dfcfb894-l5k98\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.543508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.543118    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file\") pod \"limitador-limitador-74dfcfb894-l5k98\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.544130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.544103    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file\") pod \"limitador-limitador-74dfcfb894-l5k98\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.559187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.559161    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6l2\" (UniqueName: \"kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2\") pod \"limitador-limitador-74dfcfb894-l5k98\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.649359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.649317    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:42.658565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.658535    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:42.935146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.935114    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:42.967012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:42.966911    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:42.977017 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:42.972343    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a1fcf5_4291_4a0e_b026_0f4275f236f7.slice/crio-d74f80fb44d0644841d1801c809e23a4f2477fb917cdbade60b2ece1e616ffdd WatchSource:0}: Error finding container d74f80fb44d0644841d1801c809e23a4f2477fb917cdbade60b2ece1e616ffdd: Status 404 returned error can't find the container with id d74f80fb44d0644841d1801c809e23a4f2477fb917cdbade60b2ece1e616ffdd
Nov 04 01:07:43.149292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.149247    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:43.455116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.455080    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j76hv"]
Nov 04 01:07:43.458568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.458500    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:43.468886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.468864    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fkpdm\""
Nov 04 01:07:43.470627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.470604    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j76hv"]
Nov 04 01:07:43.473208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.473148    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6" event={"ID":"517f1a3c-9ae6-4bbb-a412-22101ed0784f","Type":"ContainerStarted","Data":"3756ec54b562d53b0ea514d4f616196582c43d38c9380c46d7f995499b4cc700"}
Nov 04 01:07:43.474673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.474645    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98" event={"ID":"b4a1fcf5-4291-4a0e-b026-0f4275f236f7","Type":"ContainerStarted","Data":"d74f80fb44d0644841d1801c809e23a4f2477fb917cdbade60b2ece1e616ffdd"}
Nov 04 01:07:43.553816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.553776    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpkw\" (UniqueName: \"kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw\") pod \"authorino-57f7f8659f-j76hv\" (UID: \"1351d37e-b56b-4ceb-abe8-f04641f66900\") " pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:43.601461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.601431    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:43.654695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.654657    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpkw\" (UniqueName: \"kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw\") pod \"authorino-57f7f8659f-j76hv\" (UID: \"1351d37e-b56b-4ceb-abe8-f04641f66900\") " pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:43.666950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.666917    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpkw\" (UniqueName: \"kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw\") pod \"authorino-57f7f8659f-j76hv\" (UID: \"1351d37e-b56b-4ceb-abe8-f04641f66900\") " pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:43.776978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.776907    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:43.820876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:43.820841    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:44.054409 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:07:44.054378    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1351d37e_b56b_4ceb_abe8_f04641f66900.slice/crio-2084cc8f934484b0b904ec9edf14869ad3715702115a40d04c3c8cb16cdd49e5 WatchSource:0}: Error finding container 2084cc8f934484b0b904ec9edf14869ad3715702115a40d04c3c8cb16cdd49e5: Status 404 returned error can't find the container with id 2084cc8f934484b0b904ec9edf14869ad3715702115a40d04c3c8cb16cdd49e5
Nov 04 01:07:44.059626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:44.059575    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c488b8c84-654z6" podStartSLOduration=1.956215438 podStartE2EDuration="5.059555746s" totalImagesPullingTime="3.103340308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:43.50214235 +0000 UTC m=+251.144048620" watchObservedRunningTime="2026-11-04 01:07:44.059555746 +0000 UTC m=+251.701461996"
Nov 04 01:07:44.060507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:44.060488    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j76hv"]
Nov 04 01:07:44.447410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:44.447382    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:07:44.481305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:44.481269    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j76hv" event={"ID":"1351d37e-b56b-4ceb-abe8-f04641f66900","Type":"ContainerStarted","Data":"2084cc8f934484b0b904ec9edf14869ad3715702115a40d04c3c8cb16cdd49e5"}
Nov 04 01:07:45.489476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:45.489443    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" event={"ID":"b96933e9-892e-41b6-9b0f-9568cb44c538","Type":"ContainerStarted","Data":"ebbd0d4ad3024ff5b077a455e8f070a59da63619a52ad9308bbe35ae9c75b526"}
Nov 04 01:07:45.489801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:45.489567    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:45.511515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:45.511475    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podStartSLOduration=2.173676866 podStartE2EDuration="4.511461694s" totalImagesPullingTime="2.337784828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:45.510440074 +0000 UTC m=+253.152346323" watchObservedRunningTime="2026-11-04 01:07:45.511461694 +0000 UTC m=+253.153367947"
Nov 04 01:07:46.495940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:46.495302    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98" event={"ID":"b4a1fcf5-4291-4a0e-b026-0f4275f236f7","Type":"ContainerStarted","Data":"131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24"}
Nov 04 01:07:46.495940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:46.495463    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:46.514764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:46.514598    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98" podStartSLOduration=2.030414379 podStartE2EDuration="4.514580564s" totalImagesPullingTime="2.484166185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:46.514395548 +0000 UTC m=+254.156301814" watchObservedRunningTime="2026-11-04 01:07:46.514580564 +0000 UTC m=+254.156486816"
Nov 04 01:07:47.501888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:47.501851    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j76hv" event={"ID":"1351d37e-b56b-4ceb-abe8-f04641f66900","Type":"ContainerStarted","Data":"9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4"}
Nov 04 01:07:47.518473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:47.518423    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-j76hv" podStartSLOduration=1.635172845 podStartE2EDuration="4.51840986s" totalImagesPullingTime="2.883237015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:47.51776038 +0000 UTC m=+255.159666634" watchObservedRunningTime="2026-11-04 01:07:47.51840986 +0000 UTC m=+255.160316110"
Nov 04 01:07:48.066681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:48.066650    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:48.437582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:48.437542    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j76hv"]
Nov 04 01:07:48.445901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:48.445864    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:48.648065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:48.648027    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:48.983493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:48.983455    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:49.483461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.483423    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:49.510808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.510768    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-j76hv" podUID="1351d37e-b56b-4ceb-abe8-f04641f66900" containerName="authorino" containerID="cri-o://9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4" gracePeriod=30
Nov 04 01:07:49.681326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.681273    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:49.742765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.742712    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:49.808042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.808013    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw\" (UniqueName: \"kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw\") pod \"1351d37e-b56b-4ceb-abe8-f04641f66900\" (UID: \"1351d37e-b56b-4ceb-abe8-f04641f66900\") "
Nov 04 01:07:49.810231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.810202    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw" pod "1351d37e-b56b-4ceb-abe8-f04641f66900" (UID: "1351d37e-b56b-4ceb-abe8-f04641f66900"). InnerVolumeSpecName "kube-api-access-zmpkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:07:49.908906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:49.908877    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpkw\" (UniqueName: \"kubernetes.io/projected/1351d37e-b56b-4ceb-abe8-f04641f66900-kube-api-access-zmpkw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:07:50.518956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.518924    5797 generic.go:356] "Generic (PLEG): container finished" podID="1351d37e-b56b-4ceb-abe8-f04641f66900" containerID="9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4" exitCode=0
Nov 04 01:07:50.519155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.518975    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j76hv"
Nov 04 01:07:50.519155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.519008    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j76hv" event={"ID":"1351d37e-b56b-4ceb-abe8-f04641f66900","Type":"ContainerDied","Data":"9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4"}
Nov 04 01:07:50.519155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.519039    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j76hv" event={"ID":"1351d37e-b56b-4ceb-abe8-f04641f66900","Type":"ContainerDied","Data":"2084cc8f934484b0b904ec9edf14869ad3715702115a40d04c3c8cb16cdd49e5"}
Nov 04 01:07:50.519155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.519058    5797 scope.go:122] "RemoveContainer" containerID="9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4"
Nov 04 01:07:50.527414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.527393    5797 scope.go:122] "RemoveContainer" containerID="9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4"
Nov 04 01:07:50.527655 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:07:50.527639    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4\": container with ID starting with 9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4 not found: ID does not exist" containerID="9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4"
Nov 04 01:07:50.527715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.527663    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4"} err="failed to get container status \"9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4\": rpc error: code = NotFound desc = could not find container \"9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4\": container with ID starting with 9e59ef27764ed1d1acf9452d211783eaa50698ed7494d8a97532f15b1f6d84c4 not found: ID does not exist"
Nov 04 01:07:50.535783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.535763    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j76hv"]
Nov 04 01:07:50.537843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:50.537824    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j76hv"]
Nov 04 01:07:52.476631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:52.476597    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1351d37e-b56b-4ceb-abe8-f04641f66900" path="/var/lib/kubelet/pods/1351d37e-b56b-4ceb-abe8-f04641f66900/volumes"
Nov 04 01:07:53.478531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:53.478495    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:07:53.726483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:53.726438    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:56.250158 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:07:56.250122    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:07:56.498496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:56.498468    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:07:56.785262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:56.785231    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:56.785447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:56.785426    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98" podUID="b4a1fcf5-4291-4a0e-b026-0f4275f236f7" containerName="limitador" containerID="cri-o://131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24" gracePeriod=30
Nov 04 01:07:56.786166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:56.786130    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:57.318354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.318326    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:57.362903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.362842    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file\" (UniqueName: \"kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file\") pod \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") "
Nov 04 01:07:57.363034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.363008    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2\" (UniqueName: \"kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2\") pod \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\" (UID: \"b4a1fcf5-4291-4a0e-b026-0f4275f236f7\") "
Nov 04 01:07:57.363252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.363230    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file" pod "b4a1fcf5-4291-4a0e-b026-0f4275f236f7" (UID: "b4a1fcf5-4291-4a0e-b026-0f4275f236f7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:07:57.364863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.364839    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2" pod "b4a1fcf5-4291-4a0e-b026-0f4275f236f7" (UID: "b4a1fcf5-4291-4a0e-b026-0f4275f236f7"). InnerVolumeSpecName "kube-api-access-zr6l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:07:57.464481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.464454    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6l2\" (UniqueName: \"kubernetes.io/projected/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-kube-api-access-zr6l2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:07:57.464481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.464482    5797 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4a1fcf5-4291-4a0e-b026-0f4275f236f7-config-file\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:07:57.548111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.548076    5797 generic.go:356] "Generic (PLEG): container finished" podID="b4a1fcf5-4291-4a0e-b026-0f4275f236f7" containerID="131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24" exitCode=0
Nov 04 01:07:57.548259 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.548119    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98" event={"ID":"b4a1fcf5-4291-4a0e-b026-0f4275f236f7","Type":"ContainerDied","Data":"131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24"}
Nov 04 01:07:57.548259 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.548129    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98"
Nov 04 01:07:57.548259 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.548167    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-l5k98" event={"ID":"b4a1fcf5-4291-4a0e-b026-0f4275f236f7","Type":"ContainerDied","Data":"d74f80fb44d0644841d1801c809e23a4f2477fb917cdbade60b2ece1e616ffdd"}
Nov 04 01:07:57.548259 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.548197    5797 scope.go:122] "RemoveContainer" containerID="131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24"
Nov 04 01:07:57.557117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.557099    5797 scope.go:122] "RemoveContainer" containerID="131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24"
Nov 04 01:07:57.557393 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:07:57.557374    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24\": container with ID starting with 131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24 not found: ID does not exist" containerID="131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24"
Nov 04 01:07:57.557436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.557402    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24"} err="failed to get container status \"131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24\": rpc error: code = NotFound desc = could not find container \"131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24\": container with ID starting with 131c70f49234291cc70831cebbd7d54b4642a17b5d625f7cb3404ed683ec5f24 not found: ID does not exist"
Nov 04 01:07:57.571303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.571276    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:57.577192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:57.577173    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-l5k98"]
Nov 04 01:07:58.375561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:58.375528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:58.475560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:58.475528    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a1fcf5-4291-4a0e-b026-0f4275f236f7" path="/var/lib/kubelet/pods/b4a1fcf5-4291-4a0e-b026-0f4275f236f7/volumes"
Nov 04 01:07:58.571824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:58.571791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:58.982702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:58.982669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:07:59.192196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:07:59.192150    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:01.856724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:01.856686    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:02.072374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.072337    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:02.276208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.276176    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:02.408750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.408717    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"]
Nov 04 01:08:02.409082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.409068    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1351d37e-b56b-4ceb-abe8-f04641f66900" containerName="authorino"
Nov 04 01:08:02.409130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.409083    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1351d37e-b56b-4ceb-abe8-f04641f66900" containerName="authorino"
Nov 04 01:08:02.409130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.409095    5797 cpu_manager.go:414] "containerMap: removing container" podUID="b4a1fcf5-4291-4a0e-b026-0f4275f236f7" containerName="limitador"
Nov 04 01:08:02.409130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.409100    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a1fcf5-4291-4a0e-b026-0f4275f236f7" containerName="limitador"
Nov 04 01:08:02.409219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.409155    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1351d37e-b56b-4ceb-abe8-f04641f66900" containerName="authorino"
Nov 04 01:08:02.409219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.409163    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a1fcf5-4291-4a0e-b026-0f4275f236f7" containerName="limitador"
Nov 04 01:08:02.413518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.413500    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.417719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.417698    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v9wtr\""
Nov 04 01:08:02.418637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.418617    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 04 01:08:02.419364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.419343    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 04 01:08:02.419455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.419429    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 04 01:08:02.429320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.429300    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"]
Nov 04 01:08:02.511254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.511227    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42fv\" (UniqueName: \"kubernetes.io/projected/35506939-7454-4450-9d9b-687b5caaea77-kube-api-access-h42fv\") pod \"postgres-7cdbf6dc44-gsw2f\" (UID: \"35506939-7454-4450-9d9b-687b5caaea77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.511254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.511269    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35506939-7454-4450-9d9b-687b5caaea77-data\") pod \"postgres-7cdbf6dc44-gsw2f\" (UID: \"35506939-7454-4450-9d9b-687b5caaea77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.606395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.606313    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:02.612464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.612445    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35506939-7454-4450-9d9b-687b5caaea77-data\") pod \"postgres-7cdbf6dc44-gsw2f\" (UID: \"35506939-7454-4450-9d9b-687b5caaea77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.612541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.612529    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42fv\" (UniqueName: \"kubernetes.io/projected/35506939-7454-4450-9d9b-687b5caaea77-kube-api-access-h42fv\") pod \"postgres-7cdbf6dc44-gsw2f\" (UID: \"35506939-7454-4450-9d9b-687b5caaea77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.612821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.612804    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35506939-7454-4450-9d9b-687b5caaea77-data\") pod \"postgres-7cdbf6dc44-gsw2f\" (UID: \"35506939-7454-4450-9d9b-687b5caaea77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.624727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.624701    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42fv\" (UniqueName: \"kubernetes.io/projected/35506939-7454-4450-9d9b-687b5caaea77-kube-api-access-h42fv\") pod \"postgres-7cdbf6dc44-gsw2f\" (UID: \"35506939-7454-4450-9d9b-687b5caaea77\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.726667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.726636    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:02.995902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:02.995828    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"]
Nov 04 01:08:02.998394 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:02.998365    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35506939_7454_4450_9d9b_687b5caaea77.slice/crio-ea359e7233204b97ae2386a12bd9131e5c13ed3c92816312349250b371bac6b2 WatchSource:0}: Error finding container ea359e7233204b97ae2386a12bd9131e5c13ed3c92816312349250b371bac6b2: Status 404 returned error can't find the container with id ea359e7233204b97ae2386a12bd9131e5c13ed3c92816312349250b371bac6b2
Nov 04 01:08:03.573786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:03.573752    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" event={"ID":"35506939-7454-4450-9d9b-687b5caaea77","Type":"ContainerStarted","Data":"ea359e7233204b97ae2386a12bd9131e5c13ed3c92816312349250b371bac6b2"}
Nov 04 01:08:07.411530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:07.411497    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:07.618316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:07.618281    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:11.258485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.258371    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:11.463325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.463278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:11.610571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.610485    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" event={"ID":"35506939-7454-4450-9d9b-687b5caaea77","Type":"ContainerStarted","Data":"cbb7367064fefa917bc29cdb67c47a40e4049fa07a36d9c5f40052849eda36d5"}
Nov 04 01:08:11.610571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.610570    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:11.629383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.629333    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podStartSLOduration=1.6492744080000001 podStartE2EDuration="9.629318495s" totalImagesPullingTime="7.980044087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:11.628168348 +0000 UTC m=+279.270074613" watchObservedRunningTime="2026-11-04 01:08:11.629318495 +0000 UTC m=+279.271224744"
Nov 04 01:08:11.658593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.658565    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:11.973091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:11.973058    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:17.642814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:17.642785    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:08:20.068314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:20.068280    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:20.268482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:20.268447    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:22.581169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:22.581135    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:22.777420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:22.777383    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:25.117042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.117004    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8b95c9c66-v64v2"]
Nov 04 01:08:25.125972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.125942    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.130277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.130251    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 04 01:08:25.131371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.131307    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 04 01:08:25.131371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.131353    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 04 01:08:25.131371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.131353    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 04 01:08:25.131680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.131385    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 04 01:08:25.131891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.131860    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gx6cl\""
Nov 04 01:08:25.132648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.132627    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-v64v2"]
Nov 04 01:08:25.138191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.138158    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:25.213562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.213530    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.213730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.213584    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.213730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.213711    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddksb\" (UniqueName: \"kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.314417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.314381    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.314575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.314447    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddksb\" (UniqueName: \"kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.314575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.314494    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.317121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.317090    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.317121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.317090    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.326244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.326222    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddksb\" (UniqueName: \"kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb\") pod \"maas-controller-8b95c9c66-v64v2\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") " pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.342341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.342311    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:25.440359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.440288    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:25.470110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.470082    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-v9np2"]
Nov 04 01:08:25.473776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.473739    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.484755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.484725    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-v9np2"]
Nov 04 01:08:25.540338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.540289    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:25.616826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.616786    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.616982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.616845    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.617056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.617014    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6m6q\" (UniqueName: \"kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.715383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.715360    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-v64v2"]
Nov 04 01:08:25.717692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.717666    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.717798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.717757    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6m6q\" (UniqueName: \"kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.717846 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:25.717787    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4411d87b_003f_4a34_8306_d662009a270f.slice/crio-a75d459cf74261ffc42802051ff2dc8383042aa9b37e141e05fbad00a55471ae WatchSource:0}: Error finding container a75d459cf74261ffc42802051ff2dc8383042aa9b37e141e05fbad00a55471ae: Status 404 returned error can't find the container with id a75d459cf74261ffc42802051ff2dc8383042aa9b37e141e05fbad00a55471ae
Nov 04 01:08:25.717846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.717816    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.720266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.720245    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.720415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.720399    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.729747 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.729725    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6m6q\" (UniqueName: \"kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q\") pod \"maas-controller-5bf8c4d6cc-v9np2\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") " pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:25.788864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:25.788839    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:26.061399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:26.061375    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-v9np2"]
Nov 04 01:08:26.063703 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:26.063675    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd34cf22_51b7_4c47_969d_d8f364a35a38.slice/crio-fa683223a5953341f3a43ef660395c56a1b2ea023876ab099a86c9f5d001c7e3 WatchSource:0}: Error finding container fa683223a5953341f3a43ef660395c56a1b2ea023876ab099a86c9f5d001c7e3: Status 404 returned error can't find the container with id fa683223a5953341f3a43ef660395c56a1b2ea023876ab099a86c9f5d001c7e3
Nov 04 01:08:26.250377 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:08:26.250337    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:08:26.673063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:26.673024    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerStarted","Data":"fa683223a5953341f3a43ef660395c56a1b2ea023876ab099a86c9f5d001c7e3"}
Nov 04 01:08:26.674121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:26.674098    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-v64v2" event={"ID":"4411d87b-003f-4a34-8306-d662009a270f","Type":"ContainerStarted","Data":"a75d459cf74261ffc42802051ff2dc8383042aa9b37e141e05fbad00a55471ae"}
Nov 04 01:08:29.397650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:29.397619    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:30.694020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.693968    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerStarted","Data":"6d50afa4d6c1ee46feab2c9a1390888b49cf4169d5e40897dbf8a7566afab311"}
Nov 04 01:08:30.694465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.694096    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:30.695522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.695487    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-v64v2" event={"ID":"4411d87b-003f-4a34-8306-d662009a270f","Type":"ContainerStarted","Data":"43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5"}
Nov 04 01:08:30.695632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.695619    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:30.788142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.788079    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8b95c9c66-v64v2" podStartSLOduration=0.944601764 podStartE2EDuration="5.788061831s" totalImagesPullingTime="4.843460067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:30.787819924 +0000 UTC m=+298.429726199" watchObservedRunningTime="2026-11-04 01:08:30.788061831 +0000 UTC m=+298.429968073"
Nov 04 01:08:30.788368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.788335    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podStartSLOduration=1.298981485 podStartE2EDuration="5.788328123s" totalImagesPullingTime="4.489346638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:30.720627855 +0000 UTC m=+298.362534105" watchObservedRunningTime="2026-11-04 01:08:30.788328123 +0000 UTC m=+298.430234373"
Nov 04 01:08:30.921327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:30.921292    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:31.124845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:31.124815    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:31.990496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:31.990440    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"]
Nov 04 01:08:31.994469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:31.994445    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.006138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.006111    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"]
Nov 04 01:08:32.008036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.007977    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 04 01:08:32.008217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.008028    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 04 01:08:32.051516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.051489    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qtnvf\""
Nov 04 01:08:32.051516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.051503    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 04 01:08:32.077764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.077733    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13f0c474-26fd-4004-8b30-14358246c509-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.077924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.077787    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvnn\" (UniqueName: \"kubernetes.io/projected/13f0c474-26fd-4004-8b30-14358246c509-kube-api-access-bdvnn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.077924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.077912    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/13f0c474-26fd-4004-8b30-14358246c509-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.078043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.077946    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13f0c474-26fd-4004-8b30-14358246c509-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.160942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.160904    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:32.178766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.178737    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvnn\" (UniqueName: \"kubernetes.io/projected/13f0c474-26fd-4004-8b30-14358246c509-kube-api-access-bdvnn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.178922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.178787    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/13f0c474-26fd-4004-8b30-14358246c509-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.178922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.178813    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13f0c474-26fd-4004-8b30-14358246c509-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.178922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.178850    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13f0c474-26fd-4004-8b30-14358246c509-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.179773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.179744    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13f0c474-26fd-4004-8b30-14358246c509-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.179904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.179839    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/13f0c474-26fd-4004-8b30-14358246c509-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.181453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.181434    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13f0c474-26fd-4004-8b30-14358246c509-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.191188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.191169    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvnn\" (UniqueName: \"kubernetes.io/projected/13f0c474-26fd-4004-8b30-14358246c509-kube-api-access-bdvnn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9s6bw\" (UID: \"13f0c474-26fd-4004-8b30-14358246c509\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.308603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.308532    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:32.373248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.373175    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:32.602358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.602336    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"]
Nov 04 01:08:32.604054 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:32.604022    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f0c474_26fd_4004_8b30_14358246c509.slice/crio-5b7f6deaaddf22c033ffc2779dcc3af66c0c2325b0683c4021eeabdfac20f16b WatchSource:0}: Error finding container 5b7f6deaaddf22c033ffc2779dcc3af66c0c2325b0683c4021eeabdfac20f16b: Status 404 returned error can't find the container with id 5b7f6deaaddf22c033ffc2779dcc3af66c0c2325b0683c4021eeabdfac20f16b
Nov 04 01:08:32.705630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:32.705592    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" event={"ID":"13f0c474-26fd-4004-8b30-14358246c509","Type":"ContainerStarted","Data":"5b7f6deaaddf22c033ffc2779dcc3af66c0c2325b0683c4021eeabdfac20f16b"}
Nov 04 01:08:35.259967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.259933    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:35.309978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.309919    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mvf9p"]
Nov 04 01:08:35.314737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.314506    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.325875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.325847    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w2gqm\""
Nov 04 01:08:35.326031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.326002    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 04 01:08:35.326503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.326479    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mvf9p"]
Nov 04 01:08:35.367220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.367183    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-jr85b"]
Nov 04 01:08:35.371214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.371168    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.385406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.385350    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jr85b"]
Nov 04 01:08:35.419244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.419202    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc-cert\") pod \"kserve-controller-manager-776cc66d47-mvf9p\" (UID: \"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.419425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.419267    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlv9\" (UniqueName: \"kubernetes.io/projected/6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc-kube-api-access-rjlv9\") pod \"kserve-controller-manager-776cc66d47-mvf9p\" (UID: \"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.425216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.425188    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 04 01:08:35.425216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.425211    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6bhsh\""
Nov 04 01:08:35.425423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.425188    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 04 01:08:35.467480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.467442    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:35.521247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.521165    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-metrics-certs\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.521247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.521226    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqkht\" (UniqueName: \"kubernetes.io/projected/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-kube-api-access-tqkht\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.521481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.521365    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc-cert\") pod \"kserve-controller-manager-776cc66d47-mvf9p\" (UID: \"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.521481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.521415    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlv9\" (UniqueName: \"kubernetes.io/projected/6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc-kube-api-access-rjlv9\") pod \"kserve-controller-manager-776cc66d47-mvf9p\" (UID: \"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.521481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.521447    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-cert\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.524968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.524938    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc-cert\") pod \"kserve-controller-manager-776cc66d47-mvf9p\" (UID: \"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.546300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.546269    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlv9\" (UniqueName: \"kubernetes.io/projected/6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc-kube-api-access-rjlv9\") pod \"kserve-controller-manager-776cc66d47-mvf9p\" (UID: \"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.622450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.622404    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-metrics-certs\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.622738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.622457    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqkht\" (UniqueName: \"kubernetes.io/projected/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-kube-api-access-tqkht\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.622738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.622522    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-cert\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.633465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.633440    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:35.649938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.649910    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-cert\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.649938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.649929    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-metrics-certs\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.651598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.651571    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqkht\" (UniqueName: \"kubernetes.io/projected/8931f2f8-db63-47c5-9e05-15b4a0eb9c51-kube-api-access-tqkht\") pod \"odh-model-controller-769875db56-jr85b\" (UID: \"8931f2f8-db63-47c5-9e05-15b4a0eb9c51\") " pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.668579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.668553    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:35.692543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.692097    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:35.953845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:35.953822    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mvf9p"]
Nov 04 01:08:35.956947 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:35.956912    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6e56b0_ddd0_4e20_b6c3_4b56e51350fc.slice/crio-72fa62389f00caf0717410eeb4b07b1dca2efe8be865ce7182f66496d264c17c WatchSource:0}: Error finding container 72fa62389f00caf0717410eeb4b07b1dca2efe8be865ce7182f66496d264c17c: Status 404 returned error can't find the container with id 72fa62389f00caf0717410eeb4b07b1dca2efe8be865ce7182f66496d264c17c
Nov 04 01:08:36.025048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:36.025024    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jr85b"]
Nov 04 01:08:36.027399 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:36.027362    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8931f2f8_db63_47c5_9e05_15b4a0eb9c51.slice/crio-ae9db8cf76ae18294af0ece3316279d03dc2268452cbd1d7a36d9faa8bfe2a2c WatchSource:0}: Error finding container ae9db8cf76ae18294af0ece3316279d03dc2268452cbd1d7a36d9faa8bfe2a2c: Status 404 returned error can't find the container with id ae9db8cf76ae18294af0ece3316279d03dc2268452cbd1d7a36d9faa8bfe2a2c
Nov 04 01:08:36.754074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:36.753960    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" event={"ID":"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc","Type":"ContainerStarted","Data":"72fa62389f00caf0717410eeb4b07b1dca2efe8be865ce7182f66496d264c17c"}
Nov 04 01:08:36.755865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:36.755793    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jr85b" event={"ID":"8931f2f8-db63-47c5-9e05-15b4a0eb9c51","Type":"ContainerStarted","Data":"ae9db8cf76ae18294af0ece3316279d03dc2268452cbd1d7a36d9faa8bfe2a2c"}
Nov 04 01:08:37.058347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:37.058208    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:37.266015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:37.265910    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:37.465556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:37.465519    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:39.786158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:39.782884    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" event={"ID":"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc","Type":"ContainerStarted","Data":"778a02dcb4ffc9bd4ea192d083404f9c732dfb9e612d972ab4bdf34781d8962c"}
Nov 04 01:08:40.790906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:40.790328    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jr85b" event={"ID":"8931f2f8-db63-47c5-9e05-15b4a0eb9c51","Type":"ContainerStarted","Data":"c6d8ec71f25539eb22e2ec7b846e61bf1e781f486d8d9836cec745e2b01c95e2"}
Nov 04 01:08:40.790906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:40.790788    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:40.790906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:40.790822    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:08:40.831391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:40.831329    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podStartSLOduration=2.102138523 podStartE2EDuration="5.83131081s" totalImagesPullingTime="3.729172287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:40.810688241 +0000 UTC m=+308.452594512" watchObservedRunningTime="2026-11-04 01:08:40.83131081 +0000 UTC m=+308.473217060"
Nov 04 01:08:40.831678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:40.831651    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-jr85b" podStartSLOduration=1.491716212 podStartE2EDuration="5.831644561s" totalImagesPullingTime="4.339928349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:40.831227601 +0000 UTC m=+308.473133871" watchObservedRunningTime="2026-11-04 01:08:40.831644561 +0000 UTC m=+308.473550812"
Nov 04 01:08:41.702292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:41.702265    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:41.702475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:41.702348    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:08:41.784194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:41.784078    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-v64v2"]
Nov 04 01:08:41.819176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:41.819108    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8b95c9c66-v64v2" podUID="4411d87b-003f-4a34-8306-d662009a270f" containerName="manager" containerID="cri-o://43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5" gracePeriod=30
Nov 04 01:08:42.083841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.083813    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:42.109431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.109396    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert\") pod \"4411d87b-003f-4a34-8306-d662009a270f\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") "
Nov 04 01:08:42.109578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.109522    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs\") pod \"4411d87b-003f-4a34-8306-d662009a270f\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") "
Nov 04 01:08:42.109578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.109554    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb\" (UniqueName: \"kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb\") pod \"4411d87b-003f-4a34-8306-d662009a270f\" (UID: \"4411d87b-003f-4a34-8306-d662009a270f\") "
Nov 04 01:08:42.114309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.114257    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert" pod "4411d87b-003f-4a34-8306-d662009a270f" (UID: "4411d87b-003f-4a34-8306-d662009a270f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:08:42.114442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.114327    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb" pod "4411d87b-003f-4a34-8306-d662009a270f" (UID: "4411d87b-003f-4a34-8306-d662009a270f"). InnerVolumeSpecName "kube-api-access-ddksb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:08:42.114442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.114343    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs" pod "4411d87b-003f-4a34-8306-d662009a270f" (UID: "4411d87b-003f-4a34-8306-d662009a270f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:08:42.210503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.210465    5797 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-metrics-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:08:42.210503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.210503    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddksb\" (UniqueName: \"kubernetes.io/projected/4411d87b-003f-4a34-8306-d662009a270f-kube-api-access-ddksb\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:08:42.210503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.210520    5797 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4411d87b-003f-4a34-8306-d662009a270f-webhook-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:08:42.824805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.824723    5797 generic.go:356] "Generic (PLEG): container finished" podID="4411d87b-003f-4a34-8306-d662009a270f" containerID="43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5" exitCode=0
Nov 04 01:08:42.825261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.824823    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-v64v2" event={"ID":"4411d87b-003f-4a34-8306-d662009a270f","Type":"ContainerDied","Data":"43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5"}
Nov 04 01:08:42.825261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.824848    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-v64v2"
Nov 04 01:08:42.825261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.824888    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-v64v2" event={"ID":"4411d87b-003f-4a34-8306-d662009a270f","Type":"ContainerDied","Data":"a75d459cf74261ffc42802051ff2dc8383042aa9b37e141e05fbad00a55471ae"}
Nov 04 01:08:42.825261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.824914    5797 scope.go:122] "RemoveContainer" containerID="43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5"
Nov 04 01:08:42.834740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.834716    5797 scope.go:122] "RemoveContainer" containerID="43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5"
Nov 04 01:08:42.835090 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:08:42.835062    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5\": container with ID starting with 43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5 not found: ID does not exist" containerID="43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5"
Nov 04 01:08:42.835200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.835100    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5"} err="failed to get container status \"43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5\": rpc error: code = NotFound desc = could not find container \"43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5\": container with ID starting with 43d6cd286bb9bd4d5d7fdf18e5ec79accff8373805d99a0bc4ae5a7560f02fb5 not found: ID does not exist"
Nov 04 01:08:42.842860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.842833    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-v64v2"]
Nov 04 01:08:42.847181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:42.847157    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-v64v2"]
Nov 04 01:08:44.478770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:44.478738    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4411d87b-003f-4a34-8306-d662009a270f" path="/var/lib/kubelet/pods/4411d87b-003f-4a34-8306-d662009a270f/volumes"
Nov 04 01:08:45.392594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.392562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:45.596269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.596230    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:45.852169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.850519    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"]
Nov 04 01:08:45.852169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.851047    5797 cpu_manager.go:414] "containerMap: removing container" podUID="4411d87b-003f-4a34-8306-d662009a270f" containerName="manager"
Nov 04 01:08:45.852169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.851067    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4411d87b-003f-4a34-8306-d662009a270f" containerName="manager"
Nov 04 01:08:45.852169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.851187    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="4411d87b-003f-4a34-8306-d662009a270f" containerName="manager"
Nov 04 01:08:45.878818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.878782    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"]
Nov 04 01:08:45.878985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.878913    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:45.884029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.884003    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 04 01:08:45.884165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.884044    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 04 01:08:45.884165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.884151    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-dqwdj\""
Nov 04 01:08:45.884288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.884236    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 04 01:08:45.954178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.954149    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd717794-dd18-4bb7-8dfe-222545f12ff7-cert\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:45.954353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.954219    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd717794-dd18-4bb7-8dfe-222545f12ff7-metrics-certs\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:45.954353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.954315    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2sch\" (UniqueName: \"kubernetes.io/projected/bd717794-dd18-4bb7-8dfe-222545f12ff7-kube-api-access-g2sch\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:45.954430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:45.954405    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd717794-dd18-4bb7-8dfe-222545f12ff7-manager-config\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.055691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.055654    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd717794-dd18-4bb7-8dfe-222545f12ff7-manager-config\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.055956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.055696    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd717794-dd18-4bb7-8dfe-222545f12ff7-cert\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.055956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.055736    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd717794-dd18-4bb7-8dfe-222545f12ff7-metrics-certs\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.055956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.055761    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2sch\" (UniqueName: \"kubernetes.io/projected/bd717794-dd18-4bb7-8dfe-222545f12ff7-kube-api-access-g2sch\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.056417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.056388    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd717794-dd18-4bb7-8dfe-222545f12ff7-manager-config\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.058243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.058213    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd717794-dd18-4bb7-8dfe-222545f12ff7-metrics-certs\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.058348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.058309    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd717794-dd18-4bb7-8dfe-222545f12ff7-cert\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.082295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.082267    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2sch\" (UniqueName: \"kubernetes.io/projected/bd717794-dd18-4bb7-8dfe-222545f12ff7-kube-api-access-g2sch\") pod \"jobset-controller-manager-775c47d899-9l8hb\" (UID: \"bd717794-dd18-4bb7-8dfe-222545f12ff7\") " pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.193906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.193877    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:46.474115 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:08:46.474083    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd717794_dd18_4bb7_8dfe_222545f12ff7.slice/crio-4123e557cda95a4e77df977f4bdf4f62a2675f3b690804ef45334660977db494 WatchSource:0}: Error finding container 4123e557cda95a4e77df977f4bdf4f62a2675f3b690804ef45334660977db494: Status 404 returned error can't find the container with id 4123e557cda95a4e77df977f4bdf4f62a2675f3b690804ef45334660977db494
Nov 04 01:08:46.477254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.477228    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"]
Nov 04 01:08:46.845279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.845197    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" event={"ID":"bd717794-dd18-4bb7-8dfe-222545f12ff7","Type":"ContainerStarted","Data":"4123e557cda95a4e77df977f4bdf4f62a2675f3b690804ef45334660977db494"}
Nov 04 01:08:46.846658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.846628    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" event={"ID":"13f0c474-26fd-4004-8b30-14358246c509","Type":"ContainerStarted","Data":"e5425ac0d4e90f16ea1d24f22702cd13902fac40931733b8bdb647c0cf2965d4"}
Nov 04 01:08:46.846778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.846718    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:08:46.866099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:46.866057    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podStartSLOduration=2.329601999 podStartE2EDuration="15.866044347s" totalImagesPullingTime="13.536442348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:46.865268539 +0000 UTC m=+314.507174790" watchObservedRunningTime="2026-11-04 01:08:46.866044347 +0000 UTC m=+314.507950597"
Nov 04 01:08:50.865814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:50.865775    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" event={"ID":"bd717794-dd18-4bb7-8dfe-222545f12ff7","Type":"ContainerStarted","Data":"ec86a724fda2cd7b5c05a30fcd3f6f5edf74851e78784af3f0018ec0addc5bad"}
Nov 04 01:08:50.866286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:50.865916    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:08:50.885523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:50.885474    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podStartSLOduration=2.360379146 podStartE2EDuration="5.885460789s" totalImagesPullingTime="3.525081643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:50.884214246 +0000 UTC m=+318.526120496" watchObservedRunningTime="2026-11-04 01:08:50.885460789 +0000 UTC m=+318.527367039"
Nov 04 01:08:52.133021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:52.132957    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:08:52.827737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:52.827705    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:08:56.250481 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:08:56.250445    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:08:57.857171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:08:57.857140    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:09:01.169289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:01.169249    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:01.376193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:01.376162    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:01.876140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:01.876108    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:09:02.479671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:02.479640    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:02.682496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:02.682460    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:11.824470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:11.824432    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:09:13.218439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:13.218408    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:13.420694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:13.420660    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:26.249917 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:09:26.249874    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:09:29.453719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:29.453680    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:29.777033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:29.776924    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:29.974885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:09:29.974849    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:09:56.250577 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:09:56.250530    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:10:10.363919 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:10.363888    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:10.610093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:10.610061    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:26.249684 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:10:26.249640    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:10:26.285588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:26.285551    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:26.496483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:26.496453    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:27.037626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:27.037593    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:28.017850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:28.017817    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:29.505457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:29.505427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:30.420310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:30.420276    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:32.721236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:32.721202    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:33.819707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:33.819674    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:35.609548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.609509    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"]
Nov 04 01:10:35.613671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.613648    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.618796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.618763    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 04 01:10:35.618916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.618801    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 04 01:10:35.618916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.618814    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kbz2n\""
Nov 04 01:10:35.618916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.618836    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 04 01:10:35.623092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.623069    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"]
Nov 04 01:10:35.720854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.720820    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.720854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.720854    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dc588-8548-488f-a219-e745a8d110db-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.721105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.720898    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.721105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.720938    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v678n\" (UniqueName: \"kubernetes.io/projected/270dc588-8548-488f-a219-e745a8d110db-kube-api-access-v678n\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.721105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.721004    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.721105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.721103    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.749417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.749383    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:35.821942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.821910    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.821955    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.821970    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dc588-8548-488f-a219-e745a8d110db-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.822025    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.822044    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v678n\" (UniqueName: \"kubernetes.io/projected/270dc588-8548-488f-a219-e745a8d110db-kube-api-access-v678n\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.822066    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.822453    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.822487    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.822540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.822513    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.824262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.824240    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dc588-8548-488f-a219-e745a8d110db-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.824443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.824427    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dc588-8548-488f-a219-e745a8d110db-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.835146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.835117    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v678n\" (UniqueName: \"kubernetes.io/projected/270dc588-8548-488f-a219-e745a8d110db-kube-api-access-v678n\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qb7m9\" (UID: \"270dc588-8548-488f-a219-e745a8d110db\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:35.928832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:35.928794    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:36.201741 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:36.201720    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"]
Nov 04 01:10:36.203843 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:10:36.203815    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270dc588_8548_488f_a219_e745a8d110db.slice/crio-5208c14307525df3542fae27c8de7484437b43b6d7b8c99bc17f8ce2fa8ee4b9 WatchSource:0}: Error finding container 5208c14307525df3542fae27c8de7484437b43b6d7b8c99bc17f8ce2fa8ee4b9: Status 404 returned error can't find the container with id 5208c14307525df3542fae27c8de7484437b43b6d7b8c99bc17f8ce2fa8ee4b9
Nov 04 01:10:36.316999 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:36.316967    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" event={"ID":"270dc588-8548-488f-a219-e745a8d110db","Type":"ContainerStarted","Data":"5208c14307525df3542fae27c8de7484437b43b6d7b8c99bc17f8ce2fa8ee4b9"}
Nov 04 01:10:36.795486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:36.795306    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:38.327957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.327922    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" event={"ID":"270dc588-8548-488f-a219-e745a8d110db","Type":"ContainerStarted","Data":"7b31e284c12c5173f8511e988185146fb0637a90eecda67e1c8ee9e12e687e6e"}
Nov 04 01:10:38.349527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.349478    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podStartSLOduration=2.137367388 podStartE2EDuration="3.34946442s" totalImagesPullingTime="1.212097032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:38.347831751 +0000 UTC m=+425.989738027" watchObservedRunningTime="2026-11-04 01:10:38.34946442 +0000 UTC m=+425.991370669"
Nov 04 01:10:38.844789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.844755    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-pqxmd"]
Nov 04 01:10:38.848382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.848357    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:38.858123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.857735    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 04 01:10:38.860408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.860381    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-pqxmd"]
Nov 04 01:10:38.929801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.929769    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:38.954242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.954209    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:38.954390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.954254    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d133c42-6f8e-420e-861b-37d8e673161b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:38.954439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.954382    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-home\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:38.954439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.954410    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:38.954520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.954438    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:38.954579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:38.954557    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq28x\" (UniqueName: \"kubernetes.io/projected/5d133c42-6f8e-420e-861b-37d8e673161b-kube-api-access-fq28x\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.020731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.020676    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:10:39.055142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055107    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-home\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055142    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055174    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055235    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq28x\" (UniqueName: \"kubernetes.io/projected/5d133c42-6f8e-420e-861b-37d8e673161b-kube-api-access-fq28x\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055375    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055429    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d133c42-6f8e-420e-861b-37d8e673161b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055530    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055629    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-home\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.055685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.055641    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.057657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.057635    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d133c42-6f8e-420e-861b-37d8e673161b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.057757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.057736    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d133c42-6f8e-420e-861b-37d8e673161b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.068206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.068180    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq28x\" (UniqueName: \"kubernetes.io/projected/5d133c42-6f8e-420e-861b-37d8e673161b-kube-api-access-fq28x\") pod \"echo-simulator-kserve-64b6f8f96c-pqxmd\" (UID: \"5d133c42-6f8e-420e-861b-37d8e673161b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.164362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.164328    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:39.439573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:39.439509    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-pqxmd"]
Nov 04 01:10:39.441840 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:10:39.441809    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d133c42_6f8e_420e_861b_37d8e673161b.slice/crio-f836be2a7a616aa0a2997e43f99e3da4161ce3f01656b763cd5830e4fb04f497 WatchSource:0}: Error finding container f836be2a7a616aa0a2997e43f99e3da4161ce3f01656b763cd5830e4fb04f497: Status 404 returned error can't find the container with id f836be2a7a616aa0a2997e43f99e3da4161ce3f01656b763cd5830e4fb04f497
Nov 04 01:10:40.338642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:40.338601    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" event={"ID":"5d133c42-6f8e-420e-861b-37d8e673161b","Type":"ContainerStarted","Data":"f836be2a7a616aa0a2997e43f99e3da4161ce3f01656b763cd5830e4fb04f497"}
Nov 04 01:10:42.349765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:42.349728    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" event={"ID":"5d133c42-6f8e-420e-861b-37d8e673161b","Type":"ContainerStarted","Data":"afde52cdccc76fea96012bfc36a2b8b2d0ab36cf80429112f0320b87669c6d7c"}
Nov 04 01:10:42.370811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:42.370766    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podStartSLOduration=2.486255094 podStartE2EDuration="4.37075241s" totalImagesPullingTime="1.884497316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:42.369174876 +0000 UTC m=+430.011081147" watchObservedRunningTime="2026-11-04 01:10:42.37075241 +0000 UTC m=+430.012658656"
Nov 04 01:10:43.164569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:43.164538    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:45.635434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.635400    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"]
Nov 04 01:10:45.639497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.639473    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.648004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.647964    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4zhmf\""
Nov 04 01:10:45.656001 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.655958    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"]
Nov 04 01:10:45.719120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719089    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77q8\" (UniqueName: \"kubernetes.io/projected/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-kube-api-access-q77q8\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719129    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719152    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719252    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719312    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719365    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719383    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719403    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.719446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.719442    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820071    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820106    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820129    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77q8\" (UniqueName: \"kubernetes.io/projected/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-kube-api-access-q77q8\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820149    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820169    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820204    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820230    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820269    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820430    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820505    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820552    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820767 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820649    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.820767 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.820721    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.821177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.821157    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.822433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.822411    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.822664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.822646    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.832628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.832604    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.832738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.832683    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77q8\" (UniqueName: \"kubernetes.io/projected/c6b2d0c8-f594-4dc9-b8ff-a3dec860c112-kube-api-access-q77q8\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-x2k4g\" (UID: \"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:45.929223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.929140    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:45.941726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.941700    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:45.955785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:45.955766    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:46.238322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.238299    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"]
Nov 04 01:10:46.240613 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:10:46.240587    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b2d0c8_f594_4dc9_b8ff_a3dec860c112.slice/crio-7f05ee89d08f1ea76d3ec584d37ffcd74363dfb987008623b8d0c1c6fb924153 WatchSource:0}: Error finding container 7f05ee89d08f1ea76d3ec584d37ffcd74363dfb987008623b8d0c1c6fb924153: Status 404 returned error can't find the container with id 7f05ee89d08f1ea76d3ec584d37ffcd74363dfb987008623b8d0c1c6fb924153
Nov 04 01:10:46.242795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.242762    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:10:46.242887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.242846    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:10:46.242887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.242875    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:10:46.369932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.369899    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" event={"ID":"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112","Type":"ContainerStarted","Data":"348abddeca547c63f167a7039a3fb6fd5b42de177e7b425550c6183facad3d99"}
Nov 04 01:10:46.370150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.369963    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" event={"ID":"c6b2d0c8-f594-4dc9-b8ff-a3dec860c112","Type":"ContainerStarted","Data":"7f05ee89d08f1ea76d3ec584d37ffcd74363dfb987008623b8d0c1c6fb924153"}
Nov 04 01:10:46.381645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.381617    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:10:46.395243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.395192    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podStartSLOduration=1.39517777 podStartE2EDuration="1.39517777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:46.393959471 +0000 UTC m=+434.035865724" watchObservedRunningTime="2026-11-04 01:10:46.39517777 +0000 UTC m=+434.037084020"
Nov 04 01:10:46.966014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:46.958088    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:47.972161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:47.972117    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 04 01:10:47.972637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:47.972196    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 04 01:10:48.961475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:48.961431    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 04 01:10:48.961770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:48.961737    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 04 01:10:49.169111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:49.169065    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:49.184434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:49.184410    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:49.397917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:49.397891    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:10:49.961202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:49.961167    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 04 01:10:49.961520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:49.961491    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 04 01:10:50.966015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:50.956437    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 04 01:10:50.966015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:50.956504    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 04 01:10:51.966412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:51.966374    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 04 01:10:51.967034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:51.966980    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 04 01:10:52.956512 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:52.956472    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 04 01:10:52.956682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:52.956544    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 04 01:10:54.756590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:54.756542    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:54.757046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:54.756866    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:54.757760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:10:54.757739    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:10:56.250388 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:10:56.250350    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:11:00.445897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:00.445866    5797 generic.go:356] "Generic (PLEG): container finished" podID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerID="6d50afa4d6c1ee46feab2c9a1390888b49cf4169d5e40897dbf8a7566afab311" exitCode=1
Nov 04 01:11:00.446315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:00.445936    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerDied","Data":"6d50afa4d6c1ee46feab2c9a1390888b49cf4169d5e40897dbf8a7566afab311"}
Nov 04 01:11:00.446366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:00.446319    5797 scope.go:122] "RemoveContainer" containerID="6d50afa4d6c1ee46feab2c9a1390888b49cf4169d5e40897dbf8a7566afab311"
Nov 04 01:11:01.453722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:01.453687    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerStarted","Data":"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"}
Nov 04 01:11:01.454244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:01.453925    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:11:12.459831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:12.459753    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:11:15.691673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.691643    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"]
Nov 04 01:11:15.695445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.695420    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.708744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.708716    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"]
Nov 04 01:11:15.814779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.814740    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.814779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.814776    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749sf\" (UniqueName: \"kubernetes.io/projected/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-kube-api-access-749sf\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.814883    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.814958    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.815046    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.815068    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.815093    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.815148    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.815281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.815219    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916021    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916081    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916132    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916156    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749sf\" (UniqueName: \"kubernetes.io/projected/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-kube-api-access-749sf\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916213    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916251    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916282    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916317    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916361    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916462    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916542    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916833    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916881    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.916923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.916900    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.918574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.918547    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.918743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.918726    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.928157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.928138    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749sf\" (UniqueName: \"kubernetes.io/projected/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-kube-api-access-749sf\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:15.928234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:15.928178    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4a4e92-a390-4fc8-813d-71dff7e4e07c-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-b44x4\" (UID: \"fd4a4e92-a390-4fc8-813d-71dff7e4e07c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:16.010549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.010481    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:16.287761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.287663    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"]
Nov 04 01:11:16.289855 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:11:16.289823    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4a4e92_a390_4fc8_813d_71dff7e4e07c.slice/crio-f68aa470fed868d87e28e00cccf2bc67fbd18864c4f5adc23ce7c99bc57db97c WatchSource:0}: Error finding container f68aa470fed868d87e28e00cccf2bc67fbd18864c4f5adc23ce7c99bc57db97c: Status 404 returned error can't find the container with id f68aa470fed868d87e28e00cccf2bc67fbd18864c4f5adc23ce7c99bc57db97c
Nov 04 01:11:16.292131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.292095    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:16.292190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.292167    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:16.292225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.292194    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:16.520378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.520342    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" event={"ID":"fd4a4e92-a390-4fc8-813d-71dff7e4e07c","Type":"ContainerStarted","Data":"e71b1c55f25018ffe9419fb4c9c3d0e979c216aa5a89c231d075a6a278e0d452"}
Nov 04 01:11:16.520566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.520406    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" event={"ID":"fd4a4e92-a390-4fc8-813d-71dff7e4e07c","Type":"ContainerStarted","Data":"f68aa470fed868d87e28e00cccf2bc67fbd18864c4f5adc23ce7c99bc57db97c"}
Nov 04 01:11:16.547030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:16.546898    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podStartSLOduration=1.546880346 podStartE2EDuration="1.546880346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:16.543772762 +0000 UTC m=+464.185679037" watchObservedRunningTime="2026-11-04 01:11:16.546880346 +0000 UTC m=+464.188786598"
Nov 04 01:11:17.011632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:17.011601    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:17.012675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:17.012649    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:17.012795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:17.012697    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:18.011658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:18.011626    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:18.012063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:18.011682    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:19.011180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.011142    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:19.011348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.011199    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:19.082760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.082726    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ff97f47cd-jh7sr"]
Nov 04 01:11:19.086628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.086595    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.093671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.093631    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 04 01:11:19.094511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.093921    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fkpdm\""
Nov 04 01:11:19.094511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.094134    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ff97f47cd-jh7sr"]
Nov 04 01:11:19.145644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.145616    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f7141ac-7569-418e-9880-8f863b8211d4-tls-cert\") pod \"authorino-ff97f47cd-jh7sr\" (UID: \"9f7141ac-7569-418e-9880-8f863b8211d4\") " pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.145802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.145732    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzlj\" (UniqueName: \"kubernetes.io/projected/9f7141ac-7569-418e-9880-8f863b8211d4-kube-api-access-gdzlj\") pod \"authorino-ff97f47cd-jh7sr\" (UID: \"9f7141ac-7569-418e-9880-8f863b8211d4\") " pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.247034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.246960    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f7141ac-7569-418e-9880-8f863b8211d4-tls-cert\") pod \"authorino-ff97f47cd-jh7sr\" (UID: \"9f7141ac-7569-418e-9880-8f863b8211d4\") " pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.247239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.247052    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzlj\" (UniqueName: \"kubernetes.io/projected/9f7141ac-7569-418e-9880-8f863b8211d4-kube-api-access-gdzlj\") pod \"authorino-ff97f47cd-jh7sr\" (UID: \"9f7141ac-7569-418e-9880-8f863b8211d4\") " pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.275058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.262630    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f7141ac-7569-418e-9880-8f863b8211d4-tls-cert\") pod \"authorino-ff97f47cd-jh7sr\" (UID: \"9f7141ac-7569-418e-9880-8f863b8211d4\") " pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.275929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.275902    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzlj\" (UniqueName: \"kubernetes.io/projected/9f7141ac-7569-418e-9880-8f863b8211d4-kube-api-access-gdzlj\") pod \"authorino-ff97f47cd-jh7sr\" (UID: \"9f7141ac-7569-418e-9880-8f863b8211d4\") " pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.398835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.398804    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ff97f47cd-jh7sr"
Nov 04 01:11:19.693858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:19.693836    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ff97f47cd-jh7sr"]
Nov 04 01:11:19.696356 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:11:19.696313    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7141ac_7569_418e_9880_8f863b8211d4.slice/crio-d9a6804e11fe1648a6fcac383da7bf71756702be4f82a546a0ee2cdfa8e606ea WatchSource:0}: Error finding container d9a6804e11fe1648a6fcac383da7bf71756702be4f82a546a0ee2cdfa8e606ea: Status 404 returned error can't find the container with id d9a6804e11fe1648a6fcac383da7bf71756702be4f82a546a0ee2cdfa8e606ea
Nov 04 01:11:20.011634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:20.011552    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:20.011634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:20.011605    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:20.541960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:20.541924    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ff97f47cd-jh7sr" event={"ID":"9f7141ac-7569-418e-9880-8f863b8211d4","Type":"ContainerStarted","Data":"d9a6804e11fe1648a6fcac383da7bf71756702be4f82a546a0ee2cdfa8e606ea"}
Nov 04 01:11:21.011705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:21.011662    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:21.011884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:21.011717    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:21.558015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:21.554089    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ff97f47cd-jh7sr" event={"ID":"9f7141ac-7569-418e-9880-8f863b8211d4","Type":"ContainerStarted","Data":"bc3833bb635828c6b4699cc0ddca98e11db5ced7983b0d4f380c2388aa0ec220"}
Nov 04 01:11:21.581027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:21.580198    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-ff97f47cd-jh7sr" podStartSLOduration=1.562022351 podStartE2EDuration="2.580177403s" totalImagesPullingTime="1.018155052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:21.578346573 +0000 UTC m=+469.220252844" watchObservedRunningTime="2026-11-04 01:11:21.580177403 +0000 UTC m=+469.222083654"
Nov 04 01:11:22.011845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:22.011810    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:22.012021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:22.011873    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:22.280004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:22.279900    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:22.483354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:22.483288    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:23.012201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:23.012161    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:23.012814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:23.012767    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:24.011090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:24.011049    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 04 01:11:24.011291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:24.011107    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 04 01:11:26.012348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:26.012297    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:11:26.012751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:26.012380    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:11:26.252444 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:11:26.252405    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:11:27.054256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:27.054213    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:11:27.054768 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:27.054297    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:11:27.756142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:27.756111    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:27.756444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:27.756418    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:27.757295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:27.757275    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4"
Nov 04 01:11:28.320867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:28.320834    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:28.515238 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:28.515208    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:29.556416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:29.556368    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:30.717199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.717169    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"]
Nov 04 01:11:30.721300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.721275    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.738262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.738236    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"]
Nov 04 01:11:30.901278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901245    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901285    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901393    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901421    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901461    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901492    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901516    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkc6f\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901531    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:30.901689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:30.901652    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002377    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002421    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002469    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002487    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002505    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002547    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002580    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkc6f\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002605    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.002929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002665    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.003034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.002976    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.003134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.003073    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.003134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.003083    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.003235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.003154    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.003535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.003493    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.005049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.005025    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.005049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.005044    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.015410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.015382    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.015517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.015497    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkc6f\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-lvdcj\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.035330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.035310    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:31.315387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.315357    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"]
Nov 04 01:11:31.317390 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:11:31.317360    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6ded4e_6d49_4cda_bbb9_243a999d05e9.slice/crio-de4ced1fe9d6ee13a9ccb119e8fbcb209674e34a77235d2547c968fecc04ae10 WatchSource:0}: Error finding container de4ced1fe9d6ee13a9ccb119e8fbcb209674e34a77235d2547c968fecc04ae10: Status 404 returned error can't find the container with id de4ced1fe9d6ee13a9ccb119e8fbcb209674e34a77235d2547c968fecc04ae10
Nov 04 01:11:31.319671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.319642    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:31.319727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.319708    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:31.319760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.319736    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:31.609458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.609371    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" event={"ID":"6b6ded4e-6d49-4cda-bbb9-243a999d05e9","Type":"ContainerStarted","Data":"ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91"}
Nov 04 01:11:31.609458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.609437    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" event={"ID":"6b6ded4e-6d49-4cda-bbb9-243a999d05e9","Type":"ContainerStarted","Data":"de4ced1fe9d6ee13a9ccb119e8fbcb209674e34a77235d2547c968fecc04ae10"}
Nov 04 01:11:31.635451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:31.635391    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podStartSLOduration=1.635371649 podStartE2EDuration="1.635371649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:31.632864622 +0000 UTC m=+479.274770873" watchObservedRunningTime="2026-11-04 01:11:31.635371649 +0000 UTC m=+479.277277903"
Nov 04 01:11:32.035646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:32.035618    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:32.036721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:32.036699    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:32.036779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:32.036746    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:33.036334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:33.036303    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:33.036721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:33.036364    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:34.035676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:34.035643    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:34.035839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:34.035697    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:35.036602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:35.036565    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:35.037031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:35.036623    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:36.036517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:36.036481    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:36.036732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:36.036534    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:37.036049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:37.035981    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:37.036257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:37.036063    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:38.036611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:38.036572    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 04 01:11:38.037060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:38.036631    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 04 01:11:39.832186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:39.832150    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:39.833855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:39.833830    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:39.834812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:39.834791    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:11:45.745324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.745289    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"]
Nov 04 01:11:45.751094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.749401    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.755817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.755790    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.755829    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe939348-5f96-41ac-b733-2746eb79904d-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.755911    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.755966    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe939348-5f96-41ac-b733-2746eb79904d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.756062    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.756095    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.756245    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe939348-5f96-41ac-b733-2746eb79904d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.756293    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.756461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.756326    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ddt\" (UniqueName: \"kubernetes.io/projected/fe939348-5f96-41ac-b733-2746eb79904d-kube-api-access-44ddt\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.763238 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.763210    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"]
Nov 04 01:11:45.857232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857195    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe939348-5f96-41ac-b733-2746eb79904d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857251    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857277    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ddt\" (UniqueName: \"kubernetes.io/projected/fe939348-5f96-41ac-b733-2746eb79904d-kube-api-access-44ddt\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857315    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857425    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe939348-5f96-41ac-b733-2746eb79904d-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857511    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857544    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe939348-5f96-41ac-b733-2746eb79904d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857614    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857737    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857758    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-istio-data\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.857977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.857857    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-workload-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.858074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.858028    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe939348-5f96-41ac-b733-2746eb79904d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.858074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.858028    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-credential-socket\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.858186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.858128    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-workload-certs\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.859791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.859770    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe939348-5f96-41ac-b733-2746eb79904d-istio-envoy\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.860137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.860117    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe939348-5f96-41ac-b733-2746eb79904d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.869502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.869477    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe939348-5f96-41ac-b733-2746eb79904d-istio-token\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:45.869666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:45.869647    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ddt\" (UniqueName: \"kubernetes.io/projected/fe939348-5f96-41ac-b733-2746eb79904d-kube-api-access-44ddt\") pod \"maas-default-gateway-openshift-default-848f6bf8bc-dw6nw\" (UID: \"fe939348-5f96-41ac-b733-2746eb79904d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:46.066852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.066762    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:46.344292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.344259    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"]
Nov 04 01:11:46.346590 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:11:46.346563    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe939348_5f96_41ac_b733_2746eb79904d.slice/crio-c0713b011110eb884f56f23e894c0ac5eabbe9e1459ccaec02ac45b92bbb5dd7 WatchSource:0}: Error finding container c0713b011110eb884f56f23e894c0ac5eabbe9e1459ccaec02ac45b92bbb5dd7: Status 404 returned error can't find the container with id c0713b011110eb884f56f23e894c0ac5eabbe9e1459ccaec02ac45b92bbb5dd7
Nov 04 01:11:46.348790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.348760    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:46.348866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.348827    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:46.348908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.348874    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:11:46.683246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.683198    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" event={"ID":"fe939348-5f96-41ac-b733-2746eb79904d","Type":"ContainerStarted","Data":"4a8ee77291aff3319e2eb1b9b53020873bb6247312e1a4c2074549cc0d3d0107"}
Nov 04 01:11:46.683420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.683282    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" event={"ID":"fe939348-5f96-41ac-b733-2746eb79904d","Type":"ContainerStarted","Data":"c0713b011110eb884f56f23e894c0ac5eabbe9e1459ccaec02ac45b92bbb5dd7"}
Nov 04 01:11:46.708686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:46.708636    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podStartSLOduration=1.708620755 podStartE2EDuration="1.708620755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:46.707141169 +0000 UTC m=+494.349047431" watchObservedRunningTime="2026-11-04 01:11:46.708620755 +0000 UTC m=+494.350527004"
Nov 04 01:11:47.071076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:47.070890    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:47.073401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:47.073375    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:47.073526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:47.073424    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:48.070014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:48.067375    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:48.070014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:48.067437    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:49.070013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:49.069439    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:49.070013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:49.069516    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:50.071013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:50.070399    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:50.071013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:50.070464    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:51.068013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:51.067427    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:51.068013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:51.067492    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:52.069014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:52.068071    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:52.069014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:52.068134    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:53.067837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:53.067799    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 04 01:11:53.068034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:53.067853    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 04 01:11:54.398742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:54.398709    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:54.905942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:54.905913    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:54.906338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:54.906313    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:54.907179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:54.907160    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:11:54.931114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:54.931086    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:56.250675 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:11:56.250644    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:11:56.672434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:56.672395    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:56.877425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:56.877389    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:58.415087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:58.415052    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:11:58.621717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:11:58.621684    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:26.250073 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:12:26.250037    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:12:29.617837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:29.617800    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:31.689332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.689287    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:31.748065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.748032    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"]
Nov 04 01:12:31.752235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.752210    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.761552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.761530    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6dac99-openshift-default-dockercfg-cxxq4\""
Nov 04 01:12:31.766842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.766817    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"]
Nov 04 01:12:31.810577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810532    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810580    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810602    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810677    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810750    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810783    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810815    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810872    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.810970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.810933    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmbv\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912117    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmbv\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912181    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912207    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912234    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912256    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912300    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912331    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912370    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912401    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912673    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912814    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912847    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.912953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.912895    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.913230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.913210    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.914715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.914693    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.914938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.914919    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.923633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.923606    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:31.929178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.929159    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:31.929277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:31.929236    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmbv\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:32.054850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.054775    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"]
Nov 04 01:12:32.060452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.060430    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.068118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.068096    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:32.074130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.074100    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"]
Nov 04 01:12:32.074567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.074542    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-875528-openshift-default-dockercfg-9xdkz\""
Nov 04 01:12:32.115530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.114856    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.115530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.114906    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.115530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.114955    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.115530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.115043    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.115530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.115069    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.116135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.116096    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.116269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.116252    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.116317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.116277    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dcl\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.116403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.116386    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.148552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.148516    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:32.217484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217451    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217509    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217540    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217578    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217639    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217663    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217697    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217750    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.217867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.217774    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dcl\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.218104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.218077    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.218104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.218095    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.218273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.218151    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.218273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.218252    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.218533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.218506    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.220031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.220012    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.220215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.220196    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.229699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.229678    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.229875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.229858    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dcl\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.361456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.361364    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"]
Nov 04 01:12:32.367070 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:32.367033    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ab521b_2772_44a2_9f3d_a4acc73076dc.slice/crio-28115bb6d3ca093f859987c37fccb238c27f43d9f7ebd8971faeb7ae01c28078 WatchSource:0}: Error finding container 28115bb6d3ca093f859987c37fccb238c27f43d9f7ebd8971faeb7ae01c28078: Status 404 returned error can't find the container with id 28115bb6d3ca093f859987c37fccb238c27f43d9f7ebd8971faeb7ae01c28078
Nov 04 01:12:32.369866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.369840    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:32.370232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.370196    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:32.370308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.370288    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:32.370792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.370381    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:32.377838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.377816    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:32.660430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.660378    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:32.678801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.678775    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"]
Nov 04 01:12:32.681034 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:32.681005    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a58013_f4b6_44c6_8579_6a14505c8be5.slice/crio-2ce9c9a9633a19e4635ada2132c8e731477bebcae21223e3ec052d36ea9a3fa5 WatchSource:0}: Error finding container 2ce9c9a9633a19e4635ada2132c8e731477bebcae21223e3ec052d36ea9a3fa5: Status 404 returned error can't find the container with id 2ce9c9a9633a19e4635ada2132c8e731477bebcae21223e3ec052d36ea9a3fa5
Nov 04 01:12:32.684033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.683984    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:32.684141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.684112    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:32.684180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.684155    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:32.907818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.907772    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" event={"ID":"32a58013-f4b6-44c6-8579-6a14505c8be5","Type":"ContainerStarted","Data":"83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"}
Nov 04 01:12:32.907818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.907844    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" event={"ID":"32a58013-f4b6-44c6-8579-6a14505c8be5","Type":"ContainerStarted","Data":"2ce9c9a9633a19e4635ada2132c8e731477bebcae21223e3ec052d36ea9a3fa5"}
Nov 04 01:12:32.911845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.911757    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" event={"ID":"e9ab521b-2772-44a2-9f3d-a4acc73076dc","Type":"ContainerStarted","Data":"4d588ec3d09b5ef50f8f7768a068ce833d537f49b8ca4b16f29caa5aba12b48d"}
Nov 04 01:12:32.911845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.911817    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" event={"ID":"e9ab521b-2772-44a2-9f3d-a4acc73076dc","Type":"ContainerStarted","Data":"28115bb6d3ca093f859987c37fccb238c27f43d9f7ebd8971faeb7ae01c28078"}
Nov 04 01:12:32.932633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.932569    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podStartSLOduration=0.932550596 podStartE2EDuration="932.550596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:32.930796988 +0000 UTC m=+540.572703289" watchObservedRunningTime="2026-11-04 01:12:32.932550596 +0000 UTC m=+540.574456846"
Nov 04 01:12:32.954538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:32.954496    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podStartSLOduration=1.954482036 podStartE2EDuration="1.954482036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:32.953446435 +0000 UTC m=+540.595352710" watchObservedRunningTime="2026-11-04 01:12:32.954482036 +0000 UTC m=+540.596388285"
Nov 04 01:12:33.068440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.068397    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:33.075273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.075243    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:33.378220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.378193    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:33.383553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.383530    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:33.920112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.920084    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:33.920592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.920275    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:33.921014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.920978    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:33.921253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:33.921239    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:12:35.367724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:35.367687    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:35.649731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:35.649662    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:35.858929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:35.858889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:36.274363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:36.274329    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:36.487154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:36.487075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:36.700780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:36.700739    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:37.220143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.220083    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:37.276077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.276042    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"]
Nov 04 01:12:37.280083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.280053    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.289497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.289461    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-9902c3\""
Nov 04 01:12:37.289653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.289636    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-9902c3-dockercfg-wbzk9\""
Nov 04 01:12:37.289969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.289945    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"]
Nov 04 01:12:37.351249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.351222    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"]
Nov 04 01:12:37.354890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.354867    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.374525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.374502    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"]
Nov 04 01:12:37.376813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.376787    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.376934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.376871    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgv9\" (UniqueName: \"kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9\") pod \"payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.478194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.478104    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.478194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.478156    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.478420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.478220    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkxh\" (UniqueName: \"kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh\") pod \"payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.478420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.478299    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgv9\" (UniqueName: \"kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9\") pod \"payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.478778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.478760    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.491291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.491264    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgv9\" (UniqueName: \"kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9\") pod \"payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.505677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.505652    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:37.579596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.579564    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.579740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.579615    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkxh\" (UniqueName: \"kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh\") pod \"payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.580338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.580317    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.592925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.592896    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkxh\" (UniqueName: \"kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh\") pod \"payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.595494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.595466    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:37.669687 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.668543    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:37.900677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.900641    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"]
Nov 04 01:12:37.909478 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:37.909450    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2383056_569f_4805_8ee0_23ee3268483a.slice/crio-7a156233ec25123799d122c2b811162333b6c03e67225b4c97c0f99514a957e7 WatchSource:0}: Error finding container 7a156233ec25123799d122c2b811162333b6c03e67225b4c97c0f99514a957e7: Status 404 returned error can't find the container with id 7a156233ec25123799d122c2b811162333b6c03e67225b4c97c0f99514a957e7
Nov 04 01:12:37.911449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.911431    5797 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 01:12:37.944933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.944902    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" event={"ID":"a2383056-569f-4805-8ee0-23ee3268483a","Type":"ContainerStarted","Data":"7a156233ec25123799d122c2b811162333b6c03e67225b4c97c0f99514a957e7"}
Nov 04 01:12:37.953329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:37.953305    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"]
Nov 04 01:12:37.954985 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:37.954959    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ef37fa_243b_4c4f_a707_bf1b05287a42.slice/crio-5dfbf063fab4c0cb3cd9d85cc53e87751ca3c5bfdf29eb5599e28ee64745641b WatchSource:0}: Error finding container 5dfbf063fab4c0cb3cd9d85cc53e87751ca3c5bfdf29eb5599e28ee64745641b: Status 404 returned error can't find the container with id 5dfbf063fab4c0cb3cd9d85cc53e87751ca3c5bfdf29eb5599e28ee64745641b
Nov 04 01:12:38.020002 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.019898    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:38.227448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.227400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:38.427475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.427442    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:38.719579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.719502    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:38.784310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.784101    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"]
Nov 04 01:12:38.812391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.812275    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:38.812731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.812137    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"]
Nov 04 01:12:38.891551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.891519    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6dac99\""
Nov 04 01:12:38.891727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.891649    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6dac99-dockercfg-hkz26\""
Nov 04 01:12:38.902188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.902144    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmrz\" (UniqueName: \"kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz\") pod \"payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:38.902337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.902204    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:38.921695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.921661    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:38.959492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:38.959450    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" event={"ID":"e4ef37fa-243b-4c4f-a707-bf1b05287a42","Type":"ContainerStarted","Data":"5dfbf063fab4c0cb3cd9d85cc53e87751ca3c5bfdf29eb5599e28ee64745641b"}
Nov 04 01:12:39.010080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.009859    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmrz\" (UniqueName: \"kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz\") pod \"payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:39.010080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.009936    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:39.018018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.011301    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:39.026666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.026627    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmrz\" (UniqueName: \"kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz\") pod \"payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:39.134442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.134399    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:39.466395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.465459    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"]
Nov 04 01:12:39.479945 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:39.479909    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148e1b22_7bb1_4399_950c_66a0608a202a.slice/crio-cb566d115e7db076eec593cfcbb32a825f7237ff39f986f0bd86bbb99204add2 WatchSource:0}: Error finding container cb566d115e7db076eec593cfcbb32a825f7237ff39f986f0bd86bbb99204add2: Status 404 returned error can't find the container with id cb566d115e7db076eec593cfcbb32a825f7237ff39f986f0bd86bbb99204add2
Nov 04 01:12:39.671819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.671780    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:39.739684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.737954    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"]
Nov 04 01:12:39.743113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.743090    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.754288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.754243    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-875528\""
Nov 04 01:12:39.754421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.754319    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-875528-dockercfg-2v4jh\""
Nov 04 01:12:39.757877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.757776    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"]
Nov 04 01:12:39.824084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.824045    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbvj\" (UniqueName: \"kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj\") pod \"payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.824248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.824108    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.885345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.885306    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:39.929454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.928753    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbvj\" (UniqueName: \"kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj\") pod \"payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.929454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.928815    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.930776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.930718    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.967087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.961601    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbvj\" (UniqueName: \"kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj\") pod \"payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:39.972517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:39.972430    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" event={"ID":"148e1b22-7bb1-4399-950c-66a0608a202a","Type":"ContainerStarted","Data":"cb566d115e7db076eec593cfcbb32a825f7237ff39f986f0bd86bbb99204add2"}
Nov 04 01:12:40.066130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.066031    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:40.132982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.132936    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:40.155866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.155829    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"]
Nov 04 01:12:40.172535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.171006    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"]
Nov 04 01:12:40.172535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.171138    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.244536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.244486    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ba844c-dockercfg-pllvv\""
Nov 04 01:12:40.245016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.244961    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ba844c\""
Nov 04 01:12:40.246731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.246682    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrhl\" (UniqueName: \"kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl\") pod \"payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.246896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.246793    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.338842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.338759    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:40.348962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.348921    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrhl\" (UniqueName: \"kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl\") pod \"payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.349127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.349047    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.349871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.349845    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.363066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.363036    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrhl\" (UniqueName: \"kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl\") pod \"payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.370515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.370435    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"]
Nov 04 01:12:40.389189 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:40.389158    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb22100_7a0e_49ee_9ab4_66b2a54c34c8.slice/crio-705ebf38bfe0ce78a8cc08b8d3f69c9e2f4cfcabd1ae3fc74d0b13b8fda514c6 WatchSource:0}: Error finding container 705ebf38bfe0ce78a8cc08b8d3f69c9e2f4cfcabd1ae3fc74d0b13b8fda514c6: Status 404 returned error can't find the container with id 705ebf38bfe0ce78a8cc08b8d3f69c9e2f4cfcabd1ae3fc74d0b13b8fda514c6
Nov 04 01:12:40.492211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.491943    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:40.680539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.680478    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:40.717327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.717294    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"]
Nov 04 01:12:40.723242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.723212    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.732072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.731265    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"]
Nov 04 01:12:40.732072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.731931    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1036e8\""
Nov 04 01:12:40.755029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.755000    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"]
Nov 04 01:12:40.759769 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.759672    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:40.765203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.765175    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1036e8-dockercfg-6d8sx\""
Nov 04 01:12:40.769341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.769316    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"]
Nov 04 01:12:40.790091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.790064    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"]
Nov 04 01:12:40.796205 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:40.796175    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6c2f2e_3bc7_4f8e_8102_ad52576f8a4f.slice/crio-796ad0aef49b50892d631be1390be1f6a64cc20b8dc11e3ad2b8efe5bfa146db WatchSource:0}: Error finding container 796ad0aef49b50892d631be1390be1f6a64cc20b8dc11e3ad2b8efe5bfa146db: Status 404 returned error can't find the container with id 796ad0aef49b50892d631be1390be1f6a64cc20b8dc11e3ad2b8efe5bfa146db
Nov 04 01:12:40.854646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.854604    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.854865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.854672    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddps8\" (UniqueName: \"kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8\") pod \"payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.854865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.854726    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:40.854865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.854842    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2tg\" (UniqueName: \"kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg\") pod \"payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:40.893764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.893649    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:40.971881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.967882    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:40.971881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.968048    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2tg\" (UniqueName: \"kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg\") pod \"payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:40.971881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.968121    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.971881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.968174    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddps8\" (UniqueName: \"kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8\") pod \"payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.973557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.973026    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.973557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.973496    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:40.987332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.987288    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" event={"ID":"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8","Type":"ContainerStarted","Data":"705ebf38bfe0ce78a8cc08b8d3f69c9e2f4cfcabd1ae3fc74d0b13b8fda514c6"}
Nov 04 01:12:40.989478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.989414    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" event={"ID":"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f","Type":"ContainerStarted","Data":"796ad0aef49b50892d631be1390be1f6a64cc20b8dc11e3ad2b8efe5bfa146db"}
Nov 04 01:12:40.993166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.993103    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddps8\" (UniqueName: \"kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8\") pod \"payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:40.993360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:40.993336    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2tg\" (UniqueName: \"kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg\") pod \"payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:41.047705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:41.047502    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:41.083895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:41.083540    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:41.101752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:41.101625    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:41.380276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:41.380187    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"]
Nov 04 01:12:41.414143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:41.414116    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"]
Nov 04 01:12:41.420278 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:12:41.420211    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78d06ce_2464_4370_928b_3935d9667f66.slice/crio-73f15b44b66edea1d9930feb29b9f2f885c064c2ddf49865b654deb2e78af034 WatchSource:0}: Error finding container 73f15b44b66edea1d9930feb29b9f2f885c064c2ddf49865b654deb2e78af034: Status 404 returned error can't find the container with id 73f15b44b66edea1d9930feb29b9f2f885c064c2ddf49865b654deb2e78af034
Nov 04 01:12:41.781167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:41.781088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:42.003351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:42.003306    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" event={"ID":"c78d06ce-2464-4370-928b-3935d9667f66","Type":"ContainerStarted","Data":"73f15b44b66edea1d9930feb29b9f2f885c064c2ddf49865b654deb2e78af034"}
Nov 04 01:12:42.005665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:42.005570    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" event={"ID":"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8","Type":"ContainerStarted","Data":"81b185f816e6b318394e40293b7205ea60741ef90b000231321e12c45ad134d6"}
Nov 04 01:12:42.693821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:42.693790    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:42.897658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:42.897622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:43.013540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.013334    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" event={"ID":"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f","Type":"ContainerStarted","Data":"42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1"}
Nov 04 01:12:43.013540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.013451    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:43.015222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.015189    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" event={"ID":"148e1b22-7bb1-4399-950c-66a0608a202a","Type":"ContainerStarted","Data":"f570b845b338aff9dd32a4c3840b2258b56d7f0d74768b23d07ed1caa62c635a"}
Nov 04 01:12:43.015369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.015256    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:43.017141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.017115    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" event={"ID":"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8","Type":"ContainerStarted","Data":"66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238"}
Nov 04 01:12:43.017779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.017545    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:43.019628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.019602    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" event={"ID":"c78d06ce-2464-4370-928b-3935d9667f66","Type":"ContainerStarted","Data":"d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f"}
Nov 04 01:12:43.019728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.019691    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:43.021188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.021167    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" event={"ID":"e4ef37fa-243b-4c4f-a707-bf1b05287a42","Type":"ContainerStarted","Data":"12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb"}
Nov 04 01:12:43.021293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.021274    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:43.022751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.022731    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" event={"ID":"a2383056-569f-4805-8ee0-23ee3268483a","Type":"ContainerStarted","Data":"f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1"}
Nov 04 01:12:43.022849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.022790    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:43.024254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.024236    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" event={"ID":"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8","Type":"ContainerStarted","Data":"5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f"}
Nov 04 01:12:43.024363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.024353    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:43.032920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.032871    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podStartSLOduration=0.919003286 podStartE2EDuration="3.032858401s" totalImagesPullingTime="2.113855115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.032838518 +0000 UTC m=+550.674744792" watchObservedRunningTime="2026-11-04 01:12:43.032858401 +0000 UTC m=+550.674764669"
Nov 04 01:12:43.056293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.056185    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podStartSLOduration=1.909331859 podStartE2EDuration="6.056170383s" totalImagesPullingTime="4.146838524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.05357427 +0000 UTC m=+550.695480521" watchObservedRunningTime="2026-11-04 01:12:43.056170383 +0000 UTC m=+550.698076632"
Nov 04 01:12:43.073165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.073112    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podStartSLOduration=2.461892658 podStartE2EDuration="5.073098431s" totalImagesPullingTime="2.611205773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.071224241 +0000 UTC m=+550.713130510" watchObservedRunningTime="2026-11-04 01:12:43.073098431 +0000 UTC m=+550.715004681"
Nov 04 01:12:43.090853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.090799    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podStartSLOduration=2.063801854 podStartE2EDuration="3.090783608s" totalImagesPullingTime="1.026981754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.08862073 +0000 UTC m=+550.730527027" watchObservedRunningTime="2026-11-04 01:12:43.090783608 +0000 UTC m=+550.732689979"
Nov 04 01:12:43.108350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.108297    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podStartSLOduration=2.213875245 podStartE2EDuration="3.10828299s" totalImagesPullingTime="894.407745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.106946834 +0000 UTC m=+550.748853108" watchObservedRunningTime="2026-11-04 01:12:43.10828299 +0000 UTC m=+550.750189244"
Nov 04 01:12:43.190044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.189947    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" podStartSLOduration=2.070563348 podStartE2EDuration="6.189925353s" totalImagesPullingTime="4.119362005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.188428231 +0000 UTC m=+550.830334483" watchObservedRunningTime="2026-11-04 01:12:43.189925353 +0000 UTC m=+550.831831604"
Nov 04 01:12:43.190374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.190339    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podStartSLOduration=1.894135073 podStartE2EDuration="4.190324819s" totalImagesPullingTime="2.296189746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:43.132579632 +0000 UTC m=+550.774485883" watchObservedRunningTime="2026-11-04 01:12:43.190324819 +0000 UTC m=+550.832231064"
Nov 04 01:12:43.646735 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.646699    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:43.847814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:43.847779    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:44.397111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:44.397016    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:44.605208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:44.605174    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:45.193592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:45.193559    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:45.394762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:45.394725    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:45.890720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:45.890685    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:46.139948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:46.139912    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:46.884793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:46.884761    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:47.807934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:47.807901    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:48.537583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:48.537551    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:49.479019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:49.478791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:50.373905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:50.373869    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:50.577922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:50.577887    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:51.111060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:51.111020    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:51.321959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:51.321911    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:51.677841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:51.677806    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:52.504114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.503341    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"]
Nov 04 01:12:52.509501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.508436    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.514277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.514235    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"]
Nov 04 01:12:52.518609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.518584    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"default-dockercfg-qvhgx\""
Nov 04 01:12:52.518878 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.518862    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"simulator-w2-cffa80-kserve-self-signed-certs\""
Nov 04 01:12:52.592897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.592857    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvw9\" (UniqueName: \"kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.593106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.592929    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.593106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.592963    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.593228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.593102    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.593497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.593466    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.593584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.593530    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.610909 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.610877    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"kube-root-ca.crt\""
Nov 04 01:12:52.611053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.610979    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cffa80\"/\"openshift-service-ca.crt\""
Nov 04 01:12:52.695346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.695315    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.695825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.695360    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.695825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.695468    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvw9\" (UniqueName: \"kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.695825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.695520    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.695825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.695550    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.695825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.695615    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.696181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.696157    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.696284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.696233    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.696425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.696406    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.696672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.696652    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:52.697972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.697950    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.698752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.698732    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.707669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.707648    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvw9\" (UniqueName: \"kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9\") pod \"simulator-w2-cffa80-kserve-84f9fcff57-x6w7j\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") " pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.825829 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.825740    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:52.905884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:52.905821    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:53.144542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:53.144520    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"]
Nov 04 01:12:54.030912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.030861    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:54.040721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.040692    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:12:54.040890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.040780    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:12:54.040890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.040829    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:12:54.041811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.041632    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:12:54.041811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.041692    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:12:54.041811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.041730    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:12:54.041811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.041763    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:12:54.120675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.120096    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" event={"ID":"eb767966-ca5f-49cd-80aa-1728469ea3b1","Type":"ContainerStarted","Data":"0b6e7ab5915c45ae63427ef154ac1ae2bed0e79c86f35d01a349b5aacf7739db"}
Nov 04 01:12:54.120675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.120149    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" event={"ID":"eb767966-ca5f-49cd-80aa-1728469ea3b1","Type":"ContainerStarted","Data":"d1c8b29e61bd73418ef4d0ef6b3098bc9dac6d3e720363b36bc11ffc16f30bc9"}
Nov 04 01:12:54.245729 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.245651    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podStartSLOduration=2.245630151 podStartE2EDuration="2.245630151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:54.24345294 +0000 UTC m=+561.885359211" watchObservedRunningTime="2026-11-04 01:12:54.245630151 +0000 UTC m=+561.887536403"
Nov 04 01:12:54.833243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:54.829566    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:12:55.377017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:55.372101    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:55.586011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:55.585968    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:55.793351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:55.793270    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:56.215333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:56.215297    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:56.259749 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:12:56.259715    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:12:56.418279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:56.418200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:56.637364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:56.637322    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:57.139767 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.139679    5797 generic.go:356] "Generic (PLEG): container finished" podID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerID="83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a" exitCode=0
Nov 04 01:12:57.139952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.139760    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" event={"ID":"32a58013-f4b6-44c6-8579-6a14505c8be5","Type":"ContainerDied","Data":"83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"}
Nov 04 01:12:57.140386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.140367    5797 scope.go:122] "RemoveContainer" containerID="83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"
Nov 04 01:12:57.141827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.141799    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:57.141883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.141874    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:57.141917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.141905    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:12:57.274764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.274730    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:57.505791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:57.505694    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:58.150158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:58.150120    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" event={"ID":"32a58013-f4b6-44c6-8579-6a14505c8be5","Type":"ContainerStarted","Data":"353c3eb8a4ef9c8a69ea8ea3c4632e0c9719274de0a950b8220e0cd021a8e550"}
Nov 04 01:12:58.378242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:58.378194    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:12:58.378497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:58.378470    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 04 01:12:58.378596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:58.378522    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 04 01:12:58.585036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:58.584922    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:58.880662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:58.880632    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:12:59.379497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:59.378978    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 04 01:12:59.379497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:59.379079    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 04 01:12:59.421903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:12:59.421873    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:00.191733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.191683    5797 generic.go:356] "Generic (PLEG): container finished" podID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerID="4d588ec3d09b5ef50f8f7768a068ce833d537f49b8ca4b16f29caa5aba12b48d" exitCode=0
Nov 04 01:13:00.194096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.194054    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" event={"ID":"e9ab521b-2772-44a2-9f3d-a4acc73076dc","Type":"ContainerDied","Data":"4d588ec3d09b5ef50f8f7768a068ce833d537f49b8ca4b16f29caa5aba12b48d"}
Nov 04 01:13:00.194723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.194706    5797 scope.go:122] "RemoveContainer" containerID="4d588ec3d09b5ef50f8f7768a068ce833d537f49b8ca4b16f29caa5aba12b48d"
Nov 04 01:13:00.196975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.196944    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:13:00.197120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.197026    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:13:00.197120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.197079    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:13:00.278563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.278411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:00.379411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.379372    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 04 01:13:00.379621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.379437    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 04 01:13:00.508984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.508902    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"]
Nov 04 01:13:00.513821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.513796    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"]
Nov 04 01:13:00.514079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.514063    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.519271 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.519153    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"default-dockercfg-j4zvf\""
Nov 04 01:13:00.519387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.519274    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"simulator-w0-6dac99-kserve-self-signed-certs\""
Nov 04 01:13:00.520818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.520649    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"kube-root-ca.crt\""
Nov 04 01:13:00.520818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.520777    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6dac99\"/\"openshift-service-ca.crt\""
Nov 04 01:13:00.608850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.608819    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.609058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.609008    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.609355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.609325    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.609654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.609625    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.609830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.609807    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqln\" (UniqueName: \"kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.610059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.610041    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.675291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.675253    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:00.711503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.711463    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.711702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.711532    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.711702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.711570    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.711702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.711681    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.711883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.711720    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.711883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.711777    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqln\" (UniqueName: \"kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.712444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.712415    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.713074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.713031    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.713187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.713168    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.721972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.721904    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.736841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.736807    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.741011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.738529    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqln\" (UniqueName: \"kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln\") pod \"simulator-w0-6dac99-kserve-688f9c6d7f-xppnf\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") " pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.829583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.829492    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:00.891029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:00.885894    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:01.096068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.095047    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:01.199374 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:13:01.199271    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7ac48d_d096_4727_bcf6_02f7e33277af.slice/crio-41b9dd79179627f6b96945defc7e193414fd9f8b5b313d37352a50eac5c2d301 WatchSource:0}: Error finding container 41b9dd79179627f6b96945defc7e193414fd9f8b5b313d37352a50eac5c2d301: Status 404 returned error can't find the container with id 41b9dd79179627f6b96945defc7e193414fd9f8b5b313d37352a50eac5c2d301
Nov 04 01:13:01.203415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.202171    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"]
Nov 04 01:13:01.209505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.209473    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" event={"ID":"e9ab521b-2772-44a2-9f3d-a4acc73076dc","Type":"ContainerStarted","Data":"9b39d5d52f7079be593160b6018baeef5932dac8b2dda6754eb75f59ae4f8fba"}
Nov 04 01:13:01.379643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.379208    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 04 01:13:01.379643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.379269    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 04 01:13:01.405482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.405405    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:01.615373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.615341    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:01.827955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:01.827870    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:02.074014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.073327    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:13:02.074014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.073487    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 04 01:13:02.074014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.073530    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 04 01:13:02.220332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.220224    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" event={"ID":"ba7ac48d-d096-4727-bcf6-02f7e33277af","Type":"ContainerStarted","Data":"c58f2e988292e6f78c5f255b7f9f1b6cc705030778045a0bf2244d3ec49d8b56"}
Nov 04 01:13:02.220332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.220263    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" event={"ID":"ba7ac48d-d096-4727-bcf6-02f7e33277af","Type":"ContainerStarted","Data":"41b9dd79179627f6b96945defc7e193414fd9f8b5b313d37352a50eac5c2d301"}
Nov 04 01:13:02.244672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.244615    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podStartSLOduration=2.244594903 podStartE2EDuration="2.244594903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:02.242043913 +0000 UTC m=+569.883950185" watchObservedRunningTime="2026-11-04 01:13:02.244594903 +0000 UTC m=+569.886501152"
Nov 04 01:13:02.383018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.382408    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 04 01:13:02.383018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.382473    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 04 01:13:02.474093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.465465    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:02.827701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.827467    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:13:02.830148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.830120    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:02.843599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:02.843563    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:13:03.073744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.072405    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 04 01:13:03.073744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.072466    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 04 01:13:03.238330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.238303    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:13:03.389093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.382463    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 04 01:13:03.389093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.382556    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 04 01:13:03.654854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.654806    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:03.920405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.920308    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:13:03.920405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:03.920377    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:13:04.072383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:04.072338    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 04 01:13:04.072554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:04.072416    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 04 01:13:04.287001 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:04.286900    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:04.394183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:04.394147    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:13:04.395406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:04.395384    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:13:04.672016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:04.665400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:05.072457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:05.072355    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 04 01:13:05.072457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:05.072422    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 04 01:13:05.496700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:05.496663    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:06.069424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.069389    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 04 01:13:06.069714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.069684    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 04 01:13:06.370873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.370840    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"]
Nov 04 01:13:06.378407 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.378383    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.388451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.388420    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1036e8\"/\"openshift-service-ca.crt\""
Nov 04 01:13:06.388748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.388725    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1036e8\"/\"default-dockercfg-769nm\""
Nov 04 01:13:06.389064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.389046    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1036e8\"/\"simulator-w1-1036e8-kserve-self-signed-certs\""
Nov 04 01:13:06.389264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.389250    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1036e8\"/\"kube-root-ca.crt\""
Nov 04 01:13:06.392413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.392390    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"]
Nov 04 01:13:06.430162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.430102    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:06.516677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.516644    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.517092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.516790    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.517092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.516813    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.517092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.516832    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f9k\" (UniqueName: \"kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.517092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.516921    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.517092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.517055    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.617895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.617854    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.618471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.618430    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.618706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.618684    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.618903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.618874    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f9k\" (UniqueName: \"kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.619308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.618796    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.619535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.619504    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.619773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.619756    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.619955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.619780    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.621058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.620363    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.627621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.627569    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.629869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.629799    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.631087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.631067    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f9k\" (UniqueName: \"kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k\") pod \"simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") " pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.635030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.635003    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:06.692814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.692777    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:06.968912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:06.968882    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"]
Nov 04 01:13:06.973044 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:13:06.973011    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a49d108_2491_4c6c_808d_3b4fd1015e88.slice/crio-64a74c69e1b51e414a2bc50b83c43e3296bdf9da8c5e569ae62a2ec1cd78e71e WatchSource:0}: Error finding container 64a74c69e1b51e414a2bc50b83c43e3296bdf9da8c5e569ae62a2ec1cd78e71e: Status 404 returned error can't find the container with id 64a74c69e1b51e414a2bc50b83c43e3296bdf9da8c5e569ae62a2ec1cd78e71e
Nov 04 01:13:07.083536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.083344    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:13:07.085012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.084973    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:13:07.158033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.156957    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:07.250441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.250345    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" event={"ID":"9a49d108-2491-4c6c-808d-3b4fd1015e88","Type":"ContainerStarted","Data":"ef67371c91905bc82fa20063649d1294cb5cf9c730f0b667182fabaac0438fe5"}
Nov 04 01:13:07.250441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.250398    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" event={"ID":"9a49d108-2491-4c6c-808d-3b4fd1015e88","Type":"ContainerStarted","Data":"64a74c69e1b51e414a2bc50b83c43e3296bdf9da8c5e569ae62a2ec1cd78e71e"}
Nov 04 01:13:07.274575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.274520    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podStartSLOduration=1.27450351 podStartE2EDuration="1.27450351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:07.272067365 +0000 UTC m=+574.913973614" watchObservedRunningTime="2026-11-04 01:13:07.27450351 +0000 UTC m=+574.916409759"
Nov 04 01:13:07.693460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:07.693424    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:08.182219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:08.182185    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:08.984870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:08.984797    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:09.713831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:09.713783    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:10.407186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:10.407147    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:10.674958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:10.674868    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:10.830888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:10.830855    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:10.843674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:10.843647    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:11.176732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:11.176693    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:11.283248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:11.283221    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:13:12.092043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:12.092011    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:12.784375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:12.784345    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:13.181216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.181181    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"]
Nov 04 01:13:13.187098 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.187071    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.192198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.192175    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"kube-root-ca.crt\""
Nov 04 01:13:13.192391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.192370    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"default-dockercfg-26cpk\""
Nov 04 01:13:13.192391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.192381    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"openshift-service-ca.crt\""
Nov 04 01:13:13.192591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.192522    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"simulator-w5-875528-kserve-self-signed-certs\""
Nov 04 01:13:13.195706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.195682    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"]
Nov 04 01:13:13.200240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.200216    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45pt\" (UniqueName: \"kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.200433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.200274    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.200433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.200349    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.200433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.200382    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.200600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.200486    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.200600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.200549    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302248    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45pt\" (UniqueName: \"kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302294    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302314    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302499    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302596    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302669    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302867    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.302965 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.302935    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.303049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.303033    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.304604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.304583    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.304777 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.304761    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.314143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.314118    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45pt\" (UniqueName: \"kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt\") pod \"simulator-w5-875528-kserve-7bbfcbdb95-cj9tv\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") " pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.316953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.316931    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:13.502500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.502421    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:13.532857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.532823    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:13.779308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:13.779237    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"]
Nov 04 01:13:13.782629 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:13:13.782592    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a407479_18d4_4b04_ba29_7961fcf5f538.slice/crio-69669c641580e8bfcf882c5df45b38a8922ca646e5f1a266140533ad246f4630 WatchSource:0}: Error finding container 69669c641580e8bfcf882c5df45b38a8922ca646e5f1a266140533ad246f4630: Status 404 returned error can't find the container with id 69669c641580e8bfcf882c5df45b38a8922ca646e5f1a266140533ad246f4630
Nov 04 01:13:14.191585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:14.191536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:14.290933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:14.290900    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" event={"ID":"2a407479-18d4-4b04-ba29-7961fcf5f538","Type":"ContainerStarted","Data":"1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8"}
Nov 04 01:13:14.290933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:14.290945    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" event={"ID":"2a407479-18d4-4b04-ba29-7961fcf5f538","Type":"ContainerStarted","Data":"69669c641580e8bfcf882c5df45b38a8922ca646e5f1a266140533ad246f4630"}
Nov 04 01:13:14.317111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:14.317057    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podStartSLOduration=1.3170431759999999 podStartE2EDuration="1.317043176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:14.314616425 +0000 UTC m=+581.956522695" watchObservedRunningTime="2026-11-04 01:13:14.317043176 +0000 UTC m=+581.958949426"
Nov 04 01:13:14.503599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:14.503521    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:15.181365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:15.181321    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:15.862495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:15.862463    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:16.641630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:16.641597    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:16.693451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:16.693418    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:16.706464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:16.706442    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:17.319519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:17.319493    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:13:17.475686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:17.475640    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:17.678350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:17.678315    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:18.300740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:18.300703    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:19.063932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:19.063894    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:19.708748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:19.708719    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:20.388565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:20.388530    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:21.075734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:21.075701    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:21.837433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:21.837396    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:22.679363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:22.679320    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:23.386884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:23.386853    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:23.503166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:23.503135    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:23.515728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:23.515705    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:24.188779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:24.188738    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:24.368668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:24.368643    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:13:25.030516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:25.030479    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:25.690220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:25.690184    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:26.259468 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:13:26.259429    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:13:26.268547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:26.268515    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:26.473008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:26.472958    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:27.166334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:27.166301    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:27.989436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:27.989400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:28.692027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:28.691967    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:29.485835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:29.485801    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:29.685462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:29.685382    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:30.179444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:30.179402    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:30.467307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:30.467204    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:30.718015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:30.717919    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:30.917827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:30.917793    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:31.583266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.583217    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:31.734529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.734495    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"]
Nov 04 01:13:31.738525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.738500    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.747943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.747878    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1036e8\"/\"premium-w1-1036e8-kserve-self-signed-certs\""
Nov 04 01:13:31.751471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.751444    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"]
Nov 04 01:13:31.832056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.832025    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.832230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.832148    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.832230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.832220    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvb7\" (UniqueName: \"kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.832340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.832324    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.832379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.832352    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.832432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.832381    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.894808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.894740    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:31.934024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.933961    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934238 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934120    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934238 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934165    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvb7\" (UniqueName: \"kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934278    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934308    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934351    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934661    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934690    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.934787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.934728    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.936366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.936346    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.936539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.936523    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:31.946779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:31.946756    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvb7\" (UniqueName: \"kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7\") pod \"premium-w1-1036e8-kserve-c94f848c8-wr4tt\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") " pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:32.055503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:32.055468    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:32.341807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:32.341779    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"]
Nov 04 01:13:32.344028 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:13:32.344004    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc70573f5_5de4_4121_a727_c65799a4da86.slice/crio-c97375e7c3734c80606fe434662bef5a694e39b3997f1dc5df530a943ed3c363 WatchSource:0}: Error finding container c97375e7c3734c80606fe434662bef5a694e39b3997f1dc5df530a943ed3c363: Status 404 returned error can't find the container with id c97375e7c3734c80606fe434662bef5a694e39b3997f1dc5df530a943ed3c363
Nov 04 01:13:32.378328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:32.378301    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:32.402726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:32.402681    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" event={"ID":"c70573f5-5de4-4121-a727-c65799a4da86","Type":"ContainerStarted","Data":"c97375e7c3734c80606fe434662bef5a694e39b3997f1dc5df530a943ed3c363"}
Nov 04 01:13:33.010124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:33.010085    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:33.312622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:33.312486    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:33.410545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:33.410502    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" event={"ID":"c70573f5-5de4-4121-a727-c65799a4da86","Type":"ContainerStarted","Data":"4faf549968dcd2a67a29f135e372c2d2e7d1232e456701aceed7c0c42814ac54"}
Nov 04 01:13:33.433848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:33.433800    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podStartSLOduration=2.43378423 podStartE2EDuration="2.43378423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:33.430615059 +0000 UTC m=+601.072521328" watchObservedRunningTime="2026-11-04 01:13:33.43378423 +0000 UTC m=+601.075690479"
Nov 04 01:13:34.062428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:34.057128    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:34.097463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:34.097420    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:34.889355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:34.889317    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:35.641154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:35.641118    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:36.298055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:36.298019    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:36.976894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:36.976862    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:37.680006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:37.679957    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:38.349648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:38.349608    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:38.986715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:38.986679    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:39.671127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:39.671095    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:40.512723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:40.512686    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:41.021006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.020959    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"]
Nov 04 01:13:41.027551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.027520    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.037459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.037431    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"kube-root-ca.crt\""
Nov 04 01:13:41.037616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.037458    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"openshift-service-ca.crt\""
Nov 04 01:13:41.040942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.040784    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"default-dockercfg-458wm\""
Nov 04 01:13:41.040942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.040910    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a5115a\"/\"premium-w3-a5115a-kserve-self-signed-certs\""
Nov 04 01:13:41.041346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.041323    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"]
Nov 04 01:13:41.045063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.045036    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.045187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.045121    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.045187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.045166    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.045311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.045194    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.045311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.045228    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.045466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.045446    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4cx\" (UniqueName: \"kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.146793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.146763    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.146803    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.146822    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.146843    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.146901    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4cx\" (UniqueName: \"kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.146983    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.147327    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.147372    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.147458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.147438    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.149757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.149727    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.149893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.149739    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.167038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.162275    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4cx\" (UniqueName: \"kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx\") pod \"premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") " pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.174053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.173975    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:41.347067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.346784    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:41.507774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.507572    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:41.643547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:41.643522    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"]
Nov 04 01:13:41.644825 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:13:41.644793    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc958dc33_8c5d_41ae_8156_41ef575354cf.slice/crio-4427f30405f28275eb2ab5ba2eb809c74732930b6b1ebc4f8623233ce4e87c5f WatchSource:0}: Error finding container 4427f30405f28275eb2ab5ba2eb809c74732930b6b1ebc4f8623233ce4e87c5f: Status 404 returned error can't find the container with id 4427f30405f28275eb2ab5ba2eb809c74732930b6b1ebc4f8623233ce4e87c5f
Nov 04 01:13:42.057065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.056222    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:42.073752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.073717    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:42.177258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.177221    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:42.425523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.425466    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:42.465073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.465029    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" event={"ID":"c958dc33-8c5d-41ae-8156-41ef575354cf","Type":"ContainerStarted","Data":"32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315"}
Nov 04 01:13:42.465257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.465085    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" event={"ID":"c958dc33-8c5d-41ae-8156-41ef575354cf","Type":"ContainerStarted","Data":"4427f30405f28275eb2ab5ba2eb809c74732930b6b1ebc4f8623233ce4e87c5f"}
Nov 04 01:13:42.478901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.478863    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:13:42.484444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:42.484394    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podStartSLOduration=2.48437867 podStartE2EDuration="2.48437867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:42.484225495 +0000 UTC m=+610.126131745" watchObservedRunningTime="2026-11-04 01:13:42.48437867 +0000 UTC m=+610.126284921"
Nov 04 01:13:43.281010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:43.280185    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:43.353111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:43.353079    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:44.218688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:44.218647    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:44.922737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:44.922703    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:45.833214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:45.833180    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:46.575312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:46.575279    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:47.036889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.036854    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"]
Nov 04 01:13:47.040702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.040678    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.065475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.065447    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cffa80-openshift-default-dockercfg-s6kbq\""
Nov 04 01:13:47.069099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.069074    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"]
Nov 04 01:13:47.122758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.122723    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.122928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.122767    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.122928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.122789    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.122928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.122904    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.123046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.122941    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54s8\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.123046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.122963    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.123112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.123075    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.123197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.123181    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.123232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.123217    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.224854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.224823    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.224874    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.224972    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225054    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225083    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225120    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225159    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54s8\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225195    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225240    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225342    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225372    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225757    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.225933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.225905    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.226242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.226217    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.228039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.227983    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.228175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.228157    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.237888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.237865    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.237888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.237883    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54s8\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.354153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.354063    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:47.386503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.386459    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:47.648414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.648389    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"]
Nov 04 01:13:47.650409 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:13:47.650382    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa7e470_2922_43cd_a478_714d01ed9e64.slice/crio-52f97cd2521070597d272378c8982088b7ddfa0f70c7bb6c88e74d2526c47900 WatchSource:0}: Error finding container 52f97cd2521070597d272378c8982088b7ddfa0f70c7bb6c88e74d2526c47900: Status 404 returned error can't find the container with id 52f97cd2521070597d272378c8982088b7ddfa0f70c7bb6c88e74d2526c47900
Nov 04 01:13:47.652945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.652907    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:13:47.653064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.653014    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:13:47.653121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:47.653064    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:13:48.091725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:48.091626    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:48.578212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:48.569112    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" event={"ID":"3fa7e470-2922-43cd-a478-714d01ed9e64","Type":"ContainerStarted","Data":"5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48"}
Nov 04 01:13:48.578212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:48.569203    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" event={"ID":"3fa7e470-2922-43cd-a478-714d01ed9e64","Type":"ContainerStarted","Data":"52f97cd2521070597d272378c8982088b7ddfa0f70c7bb6c88e74d2526c47900"}
Nov 04 01:13:48.600206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:48.600143    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podStartSLOduration=1.600123225 podStartE2EDuration="1.600123225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:48.599717316 +0000 UTC m=+616.241623587" watchObservedRunningTime="2026-11-04 01:13:48.600123225 +0000 UTC m=+616.242029477"
Nov 04 01:13:48.899198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:48.899159    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:49.370012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:49.366054    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:49.370012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:49.366862    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 04 01:13:49.370012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:49.366912    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 04 01:13:49.629375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:49.629275    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:50.366013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:50.362069    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 04 01:13:50.366013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:50.362128    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 04 01:13:50.485806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:50.485362    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:51.353121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:51.353084    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:51.354681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:51.354654    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 04 01:13:51.354868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:51.354845    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 04 01:13:51.385433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:51.382224    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:51.386577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:51.386366    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:51.601968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:51.601940    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:13:52.355808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:52.355364    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 04 01:13:52.355808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:52.355426    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 04 01:13:52.416911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:52.416874    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:53.358870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:53.358414    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 04 01:13:53.358870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:53.358476    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 04 01:13:53.412602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:53.412568    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:54.405789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:54.405752    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:54.470087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:54.470050    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:54.492938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:54.492902    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:54.493128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:54.493019    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:13:55.183123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:55.183083    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:56.034561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:56.034513    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:56.250233 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:13:56.250198    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:13:56.870565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:56.870528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:57.478221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:57.478181    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:58.187075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:58.187034    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:58.879351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:58.879321    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:13:59.579759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:13:59.579727    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:00.283717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:00.283679    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:00.971899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:00.971839    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:01.694352 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.694316    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:01.795957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.795919    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"]
Nov 04 01:14:01.801970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.801940    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.811635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.811607    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"]
Nov 04 01:14:01.812107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.812086    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-9902c3-openshift-default-dockercfg-5blnq\""
Nov 04 01:14:01.943713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.943670    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.943913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.943818    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944024    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944140    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944167    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944185    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944211    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944233    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76pd\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:01.944418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:01.944258    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045229    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045282    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76pd\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045327    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045368    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045516    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045680    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045782    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045794    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045833    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045876    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045923    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.045954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.045880    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.046193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.046163    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.046703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.046680    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.048343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.048297    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.048589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.048568    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.057705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.057668    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.057805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.057773    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76pd\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.127123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.127087    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:02.457244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.457205    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"]
Nov 04 01:14:02.460308 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:02.460281    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79591ee2_8271_4d0a_b6f8_e4bac234b112.slice/crio-fdeee03385da797afcc8b89b2f0a13ace64444ee7beefbefba6988a75cb3e900 WatchSource:0}: Error finding container fdeee03385da797afcc8b89b2f0a13ace64444ee7beefbefba6988a75cb3e900: Status 404 returned error can't find the container with id fdeee03385da797afcc8b89b2f0a13ace64444ee7beefbefba6988a75cb3e900
Nov 04 01:14:02.463672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.463628    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:14:02.463782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.463737    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:14:02.463831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.463783    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:14:02.676126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.674766    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" event={"ID":"79591ee2-8271-4d0a-b6f8-e4bac234b112","Type":"ContainerStarted","Data":"acd2bd6a5abecae699ce96e4d442480b7768c07a109aa495e89aec122694799b"}
Nov 04 01:14:02.676126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.674837    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" event={"ID":"79591ee2-8271-4d0a-b6f8-e4bac234b112","Type":"ContainerStarted","Data":"fdeee03385da797afcc8b89b2f0a13ace64444ee7beefbefba6988a75cb3e900"}
Nov 04 01:14:02.700820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.700720    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:02.702582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:02.702518    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podStartSLOduration=1.702501094 podStartE2EDuration="1.702501094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:02.701925556 +0000 UTC m=+630.343831831" watchObservedRunningTime="2026-11-04 01:14:02.702501094 +0000 UTC m=+630.344407345"
Nov 04 01:14:03.128151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:03.128096    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:03.129515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:03.129490    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:03.129646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:03.129544    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:03.801012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:03.798415    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:04.128307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:04.128275    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:04.128483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:04.128327    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:04.786134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:04.786105    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:05.127756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:05.127724    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:05.128163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:05.127794    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:05.880806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:05.880767    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:06.127806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:06.127769    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:06.128313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:06.127831    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:06.987331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:06.987290    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:07.127570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:07.127526    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:07.127742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:07.127596    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:07.971585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:07.971548    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:08.127453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:08.127419    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:08.127626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:08.127471    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:08.898341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:08.898300    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:09.128208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:09.128169    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:14:09.128685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:09.128234    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:14:09.674680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:09.674645    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:10.132178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:10.132149    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:10.132650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:10.132596    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:10.133506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:10.133485    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:14:10.392677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:10.392560    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:11.074657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:11.074620    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:11.786811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:11.786781    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:12.474997 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:12.474901    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:13.177264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:13.177234    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:13.876789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:13.876744    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:14.686888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:14.686854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:15.464553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.464508    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:15.578730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.578699    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"]
Nov 04 01:14:15.582706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.582683    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.591248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.591217    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"]
Nov 04 01:14:15.592657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.592638    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:14:15.592764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.592644    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:14:15.614835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.614804    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.614946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.614925    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.615019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.615001    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65dq\" (UniqueName: \"kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.716317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.716229    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.716317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.716313    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.716816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.716462    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65dq\" (UniqueName: \"kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.716902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.716884    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.718787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.718760    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.729799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.729774    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65dq\" (UniqueName: \"kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq\") pod \"maas-api-revoke-sub-e0643cd6-sdjnf\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:15.898715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:15.898683    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:16.175920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.175895    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"]
Nov 04 01:14:16.178154 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:16.178126    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd480f38d_eb4a_40ed_93ee_31098585bdb4.slice/crio-aa615c4db71cb3623febac1d58d9e6420929d33f585108c2cf001f5d0507e6c2 WatchSource:0}: Error finding container aa615c4db71cb3623febac1d58d9e6420929d33f585108c2cf001f5d0507e6c2: Status 404 returned error can't find the container with id aa615c4db71cb3623febac1d58d9e6420929d33f585108c2cf001f5d0507e6c2
Nov 04 01:14:16.292818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.292769    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:16.619699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.619662    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"]
Nov 04 01:14:16.624820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.624796    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.633114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.632868    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a5115a-openshift-default-dockercfg-fx5s6\""
Nov 04 01:14:16.639372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.639349    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"]
Nov 04 01:14:16.728095 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728063    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728131    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728237    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728262    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728336    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728358    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728420    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728468    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf2l\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.728733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.728525    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.780302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.780271    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf" event={"ID":"d480f38d-eb4a-40ed-93ee-31098585bdb4","Type":"ContainerStarted","Data":"aa615c4db71cb3623febac1d58d9e6420929d33f585108c2cf001f5d0507e6c2"}
Nov 04 01:14:16.829699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.829654    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.829956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.829931    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830049    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830083    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830107    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830201    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830240    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf2l\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830342 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830293    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830389    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830477    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830484    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830677    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.830758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.830732    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.831072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.831050    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.832318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.832302    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.832594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.832577    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.841915 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.841896    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.842152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.842119    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf2l\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.941492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.940962    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:16.991452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:16.991417    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:17.229463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.229439    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"]
Nov 04 01:14:17.231771 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:17.231727    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bc8ba2_e063_4d0e_b12d_7a73718366c7.slice/crio-5cbf5b84402ee3a632746d7c3093c1d47903f4d8477c1ae7e9edabccd8d12641 WatchSource:0}: Error finding container 5cbf5b84402ee3a632746d7c3093c1d47903f4d8477c1ae7e9edabccd8d12641: Status 404 returned error can't find the container with id 5cbf5b84402ee3a632746d7c3093c1d47903f4d8477c1ae7e9edabccd8d12641
Nov 04 01:14:17.237038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.235333    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:14:17.237038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.235424    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:14:17.237038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.235470    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:14:17.786139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.786093    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:17.787803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.787771    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" event={"ID":"66bc8ba2-e063-4d0e-b12d-7a73718366c7","Type":"ContainerStarted","Data":"4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22"}
Nov 04 01:14:17.787924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.787816    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" event={"ID":"66bc8ba2-e063-4d0e-b12d-7a73718366c7","Type":"ContainerStarted","Data":"5cbf5b84402ee3a632746d7c3093c1d47903f4d8477c1ae7e9edabccd8d12641"}
Nov 04 01:14:17.810882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.810836    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podStartSLOduration=1.810822054 podStartE2EDuration="1.810822054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:17.809626942 +0000 UTC m=+645.451533220" watchObservedRunningTime="2026-11-04 01:14:17.810822054 +0000 UTC m=+645.452728351"
Nov 04 01:14:17.941564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:17.941519    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:18.678573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:18.678543    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:18.942655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:18.942573    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 04 01:14:18.943138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:18.942638    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 04 01:14:19.600442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.600407    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:19.603013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.602970    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"]
Nov 04 01:14:19.612879 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.612846    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"]
Nov 04 01:14:19.613050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.612941    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.693597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.693552    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455kf\" (UniqueName: \"kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.693770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.693694    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.693770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.693739    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.795193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.795157    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455kf\" (UniqueName: \"kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.795406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.795288    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.795406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.795322    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.796054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.796028    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.798175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.798149    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.807193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.807168    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455kf\" (UniqueName: \"kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf\") pod \"maas-api-revoke-sub-d4986295-nhm6f\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.928563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.928532    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:19.942036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.942009    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 04 01:14:19.942154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:19.942056    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 04 01:14:20.231713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.231689    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"]
Nov 04 01:14:20.273333 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:20.272974    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a5da41_4722_4572_aa97_b52dbc9c74c3.slice/crio-88ffc5efcedf114ac70a91d75a092d7595616cb4c62f2f2c6063b022d19c745b WatchSource:0}: Error finding container 88ffc5efcedf114ac70a91d75a092d7595616cb4c62f2f2c6063b022d19c745b: Status 404 returned error can't find the container with id 88ffc5efcedf114ac70a91d75a092d7595616cb4c62f2f2c6063b022d19c745b
Nov 04 01:14:20.393785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.393752    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:20.814630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.814546    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f" event={"ID":"b8a5da41-4722-4572-aa97-b52dbc9c74c3","Type":"ContainerStarted","Data":"5411ea46d65c3dbd73b45741fa33b5db70096ae5689c8ad3f3e1793e73e99fd1"}
Nov 04 01:14:20.814630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.814603    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f" event={"ID":"b8a5da41-4722-4572-aa97-b52dbc9c74c3","Type":"ContainerStarted","Data":"88ffc5efcedf114ac70a91d75a092d7595616cb4c62f2f2c6063b022d19c745b"}
Nov 04 01:14:20.816319 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.816285    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf" event={"ID":"d480f38d-eb4a-40ed-93ee-31098585bdb4","Type":"ContainerStarted","Data":"9628771fbf7f97c9e80fa735ac6d45f107f26c270999b3a409ae0f7bae28ed54"}
Nov 04 01:14:20.835108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.835056    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f" podStartSLOduration=1.835042444 podStartE2EDuration="1.835042444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:20.832927103 +0000 UTC m=+648.474833354" watchObservedRunningTime="2026-11-04 01:14:20.835042444 +0000 UTC m=+648.476948693"
Nov 04 01:14:20.879506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.879450    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf" podStartSLOduration=2.037466355 podStartE2EDuration="5.87943273s" totalImagesPullingTime="3.841966375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:20.855524546 +0000 UTC m=+648.497430823" watchObservedRunningTime="2026-11-04 01:14:20.87943273 +0000 UTC m=+648.521338974"
Nov 04 01:14:20.941896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.941864    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 04 01:14:20.942111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:20.941917    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 04 01:14:21.141141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.141108    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:21.825338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.825307    5797 generic.go:356] "Generic (PLEG): container finished" podID="b8a5da41-4722-4572-aa97-b52dbc9c74c3" containerID="5411ea46d65c3dbd73b45741fa33b5db70096ae5689c8ad3f3e1793e73e99fd1" exitCode=0
Nov 04 01:14:21.825749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.825385    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f" event={"ID":"b8a5da41-4722-4572-aa97-b52dbc9c74c3","Type":"ContainerDied","Data":"5411ea46d65c3dbd73b45741fa33b5db70096ae5689c8ad3f3e1793e73e99fd1"}
Nov 04 01:14:21.827236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.827203    5797 generic.go:356] "Generic (PLEG): container finished" podID="d480f38d-eb4a-40ed-93ee-31098585bdb4" containerID="9628771fbf7f97c9e80fa735ac6d45f107f26c270999b3a409ae0f7bae28ed54" exitCode=0
Nov 04 01:14:21.827376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.827254    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf" event={"ID":"d480f38d-eb4a-40ed-93ee-31098585bdb4","Type":"ContainerDied","Data":"9628771fbf7f97c9e80fa735ac6d45f107f26c270999b3a409ae0f7bae28ed54"}
Nov 04 01:14:21.878970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.878939    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:21.941812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.941777    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 04 01:14:21.942027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:21.941831    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 04 01:14:22.611929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:22.611900    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:22.941841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:22.941813    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 04 01:14:22.942322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:22.941873    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 04 01:14:23.083124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.083095    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:23.086303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.086279    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:23.134502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.134472    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca\") pod \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") "
Nov 04 01:14:23.134659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.134525    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca\") pod \"d480f38d-eb4a-40ed-93ee-31098585bdb4\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") "
Nov 04 01:14:23.134736 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.134713    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq\" (UniqueName: \"kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq\") pod \"d480f38d-eb4a-40ed-93ee-31098585bdb4\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") "
Nov 04 01:14:23.134881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.134859    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca\") pod \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") "
Nov 04 01:14:23.135042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.134945    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf\" (UniqueName: \"kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf\") pod \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\" (UID: \"b8a5da41-4722-4572-aa97-b52dbc9c74c3\") "
Nov 04 01:14:23.135115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.135057    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca\") pod \"d480f38d-eb4a-40ed-93ee-31098585bdb4\" (UID: \"d480f38d-eb4a-40ed-93ee-31098585bdb4\") "
Nov 04 01:14:23.135454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.135425    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca" pod "b8a5da41-4722-4572-aa97-b52dbc9c74c3" (UID: "b8a5da41-4722-4572-aa97-b52dbc9c74c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:23.135628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.135605    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:23.136029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.135980    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca" pod "d480f38d-eb4a-40ed-93ee-31098585bdb4" (UID: "d480f38d-eb4a-40ed-93ee-31098585bdb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:23.137070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.137038    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca" pod "d480f38d-eb4a-40ed-93ee-31098585bdb4" (UID: "d480f38d-eb4a-40ed-93ee-31098585bdb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:23.137315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.137296    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq" pod "d480f38d-eb4a-40ed-93ee-31098585bdb4" (UID: "d480f38d-eb4a-40ed-93ee-31098585bdb4"). InnerVolumeSpecName "kube-api-access-k65dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:23.137464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.137446    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca" pod "b8a5da41-4722-4572-aa97-b52dbc9c74c3" (UID: "b8a5da41-4722-4572-aa97-b52dbc9c74c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:23.137775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.137754    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf" pod "b8a5da41-4722-4572-aa97-b52dbc9c74c3" (UID: "b8a5da41-4722-4572-aa97-b52dbc9c74c3"). InnerVolumeSpecName "kube-api-access-455kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:23.237556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.237521    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455kf\" (UniqueName: \"kubernetes.io/projected/b8a5da41-4722-4572-aa97-b52dbc9c74c3-kube-api-access-455kf\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:23.237556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.237552    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:23.237556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.237563    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8a5da41-4722-4572-aa97-b52dbc9c74c3-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:23.237837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.237574    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d480f38d-eb4a-40ed-93ee-31098585bdb4-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:23.237837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.237583    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65dq\" (UniqueName: \"kubernetes.io/projected/d480f38d-eb4a-40ed-93ee-31098585bdb4-kube-api-access-k65dq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:23.478862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.478769    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:23.842677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.842578    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"
Nov 04 01:14:23.842677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.842596    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f" event={"ID":"b8a5da41-4722-4572-aa97-b52dbc9c74c3","Type":"ContainerDied","Data":"88ffc5efcedf114ac70a91d75a092d7595616cb4c62f2f2c6063b022d19c745b"}
Nov 04 01:14:23.842677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.842653    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ffc5efcedf114ac70a91d75a092d7595616cb4c62f2f2c6063b022d19c745b"
Nov 04 01:14:23.844218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.844189    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"
Nov 04 01:14:23.844338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.844224    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf" event={"ID":"d480f38d-eb4a-40ed-93ee-31098585bdb4","Type":"ContainerDied","Data":"aa615c4db71cb3623febac1d58d9e6420929d33f585108c2cf001f5d0507e6c2"}
Nov 04 01:14:23.844338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:23.844268    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa615c4db71cb3623febac1d58d9e6420929d33f585108c2cf001f5d0507e6c2"
Nov 04 01:14:24.094539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:24.094460    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:24.095302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:24.095264    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:24.096206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:24.096179    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:14:24.274051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:24.273985    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:24.917828 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:24.917790    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:25.108763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:25.108729    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:25.768526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:25.768480    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:26.253075 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:14:26.253037    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:14:26.438028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:26.437997    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:27.179881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.179844    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:27.839466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.839431    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"]
Nov 04 01:14:27.839937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.839924    5797 cpu_manager.go:414] "containerMap: removing container" podUID="b8a5da41-4722-4572-aa97-b52dbc9c74c3" containerName="revoke-keys"
Nov 04 01:14:27.839981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.839938    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a5da41-4722-4572-aa97-b52dbc9c74c3" containerName="revoke-keys"
Nov 04 01:14:27.839981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.839964    5797 cpu_manager.go:414] "containerMap: removing container" podUID="d480f38d-eb4a-40ed-93ee-31098585bdb4" containerName="revoke-keys"
Nov 04 01:14:27.839981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.839969    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d480f38d-eb4a-40ed-93ee-31098585bdb4" containerName="revoke-keys"
Nov 04 01:14:27.840100 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.840090    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="d480f38d-eb4a-40ed-93ee-31098585bdb4" containerName="revoke-keys"
Nov 04 01:14:27.840135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.840105    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a5da41-4722-4572-aa97-b52dbc9c74c3" containerName="revoke-keys"
Nov 04 01:14:27.843785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.843770    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:27.852733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.852698    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:14:27.853633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.853600    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:14:27.877086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.856406    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"]
Nov 04 01:14:27.994131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.994101    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:27.994555 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.994530    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rn4d\" (UniqueName: \"kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:27.994694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.994673    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:27.994798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:27.994774    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.095793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.095701    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn4d\" (UniqueName: \"kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.095967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.095933    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.096070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.096017    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.096831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.096798    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.098506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.098478    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.111011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.110322    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn4d\" (UniqueName: \"kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d\") pod \"maas-api-revoke-sub-2fa80e14-6xvt8\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.161875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.161841    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:28.487504 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:28.487465    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80281262_60a2_416f_a325_5ffdbe9a4cf1.slice/crio-351ddc6ae5f87bc399f6a5c4266df48fd619ecdd38b6bae5216665a05d52f561 WatchSource:0}: Error finding container 351ddc6ae5f87bc399f6a5c4266df48fd619ecdd38b6bae5216665a05d52f561: Status 404 returned error can't find the container with id 351ddc6ae5f87bc399f6a5c4266df48fd619ecdd38b6bae5216665a05d52f561
Nov 04 01:14:28.491280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.491259    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"]
Nov 04 01:14:28.794371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.794335    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:28.878640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.878422    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8" event={"ID":"80281262-60a2-416f-a325-5ffdbe9a4cf1","Type":"ContainerStarted","Data":"c5e214f38e7e5525b60c5cd326520d0ea3677ef0a85af8b51bfa79fd6212b699"}
Nov 04 01:14:28.878640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.878477    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8" event={"ID":"80281262-60a2-416f-a325-5ffdbe9a4cf1","Type":"ContainerStarted","Data":"351ddc6ae5f87bc399f6a5c4266df48fd619ecdd38b6bae5216665a05d52f561"}
Nov 04 01:14:28.896945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:28.896899    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8" podStartSLOduration=1.8968853989999999 podStartE2EDuration="1.896885399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:28.896213081 +0000 UTC m=+656.538119350" watchObservedRunningTime="2026-11-04 01:14:28.896885399 +0000 UTC m=+656.538791648"
Nov 04 01:14:29.563753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:29.563724    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:29.761199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:29.761143    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:29.888231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:29.888197    5797 generic.go:356] "Generic (PLEG): container finished" podID="80281262-60a2-416f-a325-5ffdbe9a4cf1" containerID="c5e214f38e7e5525b60c5cd326520d0ea3677ef0a85af8b51bfa79fd6212b699" exitCode=0
Nov 04 01:14:29.889192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:29.889160    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8" event={"ID":"80281262-60a2-416f-a325-5ffdbe9a4cf1","Type":"ContainerDied","Data":"c5e214f38e7e5525b60c5cd326520d0ea3677ef0a85af8b51bfa79fd6212b699"}
Nov 04 01:14:30.389385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:30.389351    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:30.775606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:30.775526    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:31.051660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.051623    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:31.138329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.138297    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d\" (UniqueName: \"kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d\") pod \"80281262-60a2-416f-a325-5ffdbe9a4cf1\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") "
Nov 04 01:14:31.138533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.138458    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca\") pod \"80281262-60a2-416f-a325-5ffdbe9a4cf1\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") "
Nov 04 01:14:31.138533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.138496    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca\") pod \"80281262-60a2-416f-a325-5ffdbe9a4cf1\" (UID: \"80281262-60a2-416f-a325-5ffdbe9a4cf1\") "
Nov 04 01:14:31.138891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.138862    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca" pod "80281262-60a2-416f-a325-5ffdbe9a4cf1" (UID: "80281262-60a2-416f-a325-5ffdbe9a4cf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:31.139111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.139082    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:31.140542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.140521    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d" pod "80281262-60a2-416f-a325-5ffdbe9a4cf1" (UID: "80281262-60a2-416f-a325-5ffdbe9a4cf1"). InnerVolumeSpecName "kube-api-access-7rn4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:31.140622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.140588    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca" pod "80281262-60a2-416f-a325-5ffdbe9a4cf1" (UID: "80281262-60a2-416f-a325-5ffdbe9a4cf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:31.239631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.239598    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:31.239631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.239625    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80281262-60a2-416f-a325-5ffdbe9a4cf1-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:31.239631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.239636    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rn4d\" (UniqueName: \"kubernetes.io/projected/80281262-60a2-416f-a325-5ffdbe9a4cf1-kube-api-access-7rn4d\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:31.883594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.883494    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"]
Nov 04 01:14:31.884372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.884341    5797 cpu_manager.go:414] "containerMap: removing container" podUID="80281262-60a2-416f-a325-5ffdbe9a4cf1" containerName="revoke-keys"
Nov 04 01:14:31.884372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.884363    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80281262-60a2-416f-a325-5ffdbe9a4cf1" containerName="revoke-keys"
Nov 04 01:14:31.884564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.884463    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="80281262-60a2-416f-a325-5ffdbe9a4cf1" containerName="revoke-keys"
Nov 04 01:14:31.887926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.887904    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:31.896880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.896851    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"]
Nov 04 01:14:31.904264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.904215    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8" event={"ID":"80281262-60a2-416f-a325-5ffdbe9a4cf1","Type":"ContainerDied","Data":"351ddc6ae5f87bc399f6a5c4266df48fd619ecdd38b6bae5216665a05d52f561"}
Nov 04 01:14:31.904392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.904329    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351ddc6ae5f87bc399f6a5c4266df48fd619ecdd38b6bae5216665a05d52f561"
Nov 04 01:14:31.904453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.904435    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"
Nov 04 01:14:31.990914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:31.990883    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:32.058692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.058652    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.059113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.058734    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.059113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.058920    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wpq\" (UniqueName: \"kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.160764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.160675    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.161582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.161554    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.161870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.161848    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wpq\" (UniqueName: \"kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.162567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.162537    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.165650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.165625    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.173947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.173927    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wpq\" (UniqueName: \"kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq\") pod \"maas-api-revoke-sub-4af3eca9-9b7bx\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.210083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.210059    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:32.499969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.499940    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"]
Nov 04 01:14:32.501866 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:32.501839    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15154b60_bc67_4d99_b399_ae2dbb675f2c.slice/crio-e789a2a1b908d8f91f5e9a1dd89243633218af930db4305466dba8bc05a12c98 WatchSource:0}: Error finding container e789a2a1b908d8f91f5e9a1dd89243633218af930db4305466dba8bc05a12c98: Status 404 returned error can't find the container with id e789a2a1b908d8f91f5e9a1dd89243633218af930db4305466dba8bc05a12c98
Nov 04 01:14:32.695668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.695634    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"]
Nov 04 01:14:32.700397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.700377    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.708039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.707899    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"]
Nov 04 01:14:32.796372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.796284    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:32.874270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.874236    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.874447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.874366    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qzw\" (UniqueName: \"kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.874551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.874529    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.912044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.912014    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx" event={"ID":"15154b60-bc67-4d99-b399-ae2dbb675f2c","Type":"ContainerStarted","Data":"e789a2a1b908d8f91f5e9a1dd89243633218af930db4305466dba8bc05a12c98"}
Nov 04 01:14:32.975846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.975814    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qzw\" (UniqueName: \"kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.976015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.975981    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.976115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.976101    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.976757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.976729    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.978348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.978323    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:32.987882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:32.987861    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qzw\" (UniqueName: \"kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw\") pod \"maas-api-revoke-sub-99973c09-47txl\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:33.016069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.016039    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:33.298394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.298367    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"]
Nov 04 01:14:33.299789 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:33.299763    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abda11a_e17f_421d_85fe_c40fda1026cc.slice/crio-1638e88ba9849b4d0c8c71ef2af26e6b94635aa01242b5625d04213998261d8c WatchSource:0}: Error finding container 1638e88ba9849b4d0c8c71ef2af26e6b94635aa01242b5625d04213998261d8c: Status 404 returned error can't find the container with id 1638e88ba9849b4d0c8c71ef2af26e6b94635aa01242b5625d04213998261d8c
Nov 04 01:14:33.598931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.598850    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:33.848338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.848303    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"]
Nov 04 01:14:33.852492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.852432    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:33.863921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.863889    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"]
Nov 04 01:14:33.928209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.928169    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl" event={"ID":"0abda11a-e17f-421d-85fe-c40fda1026cc","Type":"ContainerStarted","Data":"1b81a980cc46e82ed753ea123e152eef19167345bea8bb3a72099f82fd915e20"}
Nov 04 01:14:33.928399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.928236    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl" event={"ID":"0abda11a-e17f-421d-85fe-c40fda1026cc","Type":"ContainerStarted","Data":"1638e88ba9849b4d0c8c71ef2af26e6b94635aa01242b5625d04213998261d8c"}
Nov 04 01:14:33.930060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.930031    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx" event={"ID":"15154b60-bc67-4d99-b399-ae2dbb675f2c","Type":"ContainerDied","Data":"548ed26f398bb1ce58a7c81ac55b0fea409793d97f99fd680c5b5cbd35a72acf"}
Nov 04 01:14:33.930181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.929980    5797 generic.go:356] "Generic (PLEG): container finished" podID="15154b60-bc67-4d99-b399-ae2dbb675f2c" containerID="548ed26f398bb1ce58a7c81ac55b0fea409793d97f99fd680c5b5cbd35a72acf" exitCode=0
Nov 04 01:14:33.947104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.947060    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl" podStartSLOduration=1.9470467569999999 podStartE2EDuration="1.947046757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:33.946152802 +0000 UTC m=+661.588059074" watchObservedRunningTime="2026-11-04 01:14:33.947046757 +0000 UTC m=+661.588952997"
Nov 04 01:14:33.992341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.992303    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljj5p\" (UniqueName: \"kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:33.992513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.992490    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:33.992557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:33.992525    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.093632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.093591    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljj5p\" (UniqueName: \"kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.093791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.093763    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.093843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.093794    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.094409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.094383    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.096192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.096171    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.106305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.106233    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljj5p\" (UniqueName: \"kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p\") pod \"maas-api-revoke-sub-4c3b6d48-f7xmq\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.179116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.179087    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:34.455955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.455918    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"]
Nov 04 01:14:34.461712 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:34.461684    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3be2608_3dbe_4276_b598_7e3f21d0e420.slice/crio-228a4a3c7cf34e59a60a6c63ed0c7dd6e87fa58220c1d8487e93f1a6e9b31a23 WatchSource:0}: Error finding container 228a4a3c7cf34e59a60a6c63ed0c7dd6e87fa58220c1d8487e93f1a6e9b31a23: Status 404 returned error can't find the container with id 228a4a3c7cf34e59a60a6c63ed0c7dd6e87fa58220c1d8487e93f1a6e9b31a23
Nov 04 01:14:34.470503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.470478    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:34.937905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.937871    5797 generic.go:356] "Generic (PLEG): container finished" podID="0abda11a-e17f-421d-85fe-c40fda1026cc" containerID="1b81a980cc46e82ed753ea123e152eef19167345bea8bb3a72099f82fd915e20" exitCode=0
Nov 04 01:14:34.938115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.937944    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl" event={"ID":"0abda11a-e17f-421d-85fe-c40fda1026cc","Type":"ContainerDied","Data":"1b81a980cc46e82ed753ea123e152eef19167345bea8bb3a72099f82fd915e20"}
Nov 04 01:14:34.939745 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.939718    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq" event={"ID":"e3be2608-3dbe-4276-b598-7e3f21d0e420","Type":"ContainerStarted","Data":"a0045628e60594bfec193dbc679c4cae2573c977a30557b67f45f94499c90adc"}
Nov 04 01:14:34.939854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:34.939770    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq" event={"ID":"e3be2608-3dbe-4276-b598-7e3f21d0e420","Type":"ContainerStarted","Data":"228a4a3c7cf34e59a60a6c63ed0c7dd6e87fa58220c1d8487e93f1a6e9b31a23"}
Nov 04 01:14:35.083325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.083291    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:35.105150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.104711    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq" podStartSLOduration=2.104690644 podStartE2EDuration="2.104690644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:34.976159441 +0000 UTC m=+662.618065711" watchObservedRunningTime="2026-11-04 01:14:35.104690644 +0000 UTC m=+662.746596896"
Nov 04 01:14:35.206634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.206606    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq\" (UniqueName: \"kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq\") pod \"15154b60-bc67-4d99-b399-ae2dbb675f2c\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") "
Nov 04 01:14:35.206810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.206674    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca\") pod \"15154b60-bc67-4d99-b399-ae2dbb675f2c\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") "
Nov 04 01:14:35.206810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.206782    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca\") pod \"15154b60-bc67-4d99-b399-ae2dbb675f2c\" (UID: \"15154b60-bc67-4d99-b399-ae2dbb675f2c\") "
Nov 04 01:14:35.207162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.207132    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca" pod "15154b60-bc67-4d99-b399-ae2dbb675f2c" (UID: "15154b60-bc67-4d99-b399-ae2dbb675f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:35.208836 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.208812    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq" pod "15154b60-bc67-4d99-b399-ae2dbb675f2c" (UID: "15154b60-bc67-4d99-b399-ae2dbb675f2c"). InnerVolumeSpecName "kube-api-access-d6wpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:35.208942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.208818    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca" pod "15154b60-bc67-4d99-b399-ae2dbb675f2c" (UID: "15154b60-bc67-4d99-b399-ae2dbb675f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:35.305675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.305637    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:35.308161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.308138    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:35.308255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.308163    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6wpq\" (UniqueName: \"kubernetes.io/projected/15154b60-bc67-4d99-b399-ae2dbb675f2c-kube-api-access-d6wpq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:35.308255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.308173    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15154b60-bc67-4d99-b399-ae2dbb675f2c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:35.954096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.949492    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"
Nov 04 01:14:35.954096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.949497    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx" event={"ID":"15154b60-bc67-4d99-b399-ae2dbb675f2c","Type":"ContainerDied","Data":"e789a2a1b908d8f91f5e9a1dd89243633218af930db4305466dba8bc05a12c98"}
Nov 04 01:14:35.954096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.949556    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e789a2a1b908d8f91f5e9a1dd89243633218af930db4305466dba8bc05a12c98"
Nov 04 01:14:35.956219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.956191    5797 generic.go:356] "Generic (PLEG): container finished" podID="e3be2608-3dbe-4276-b598-7e3f21d0e420" containerID="a0045628e60594bfec193dbc679c4cae2573c977a30557b67f45f94499c90adc" exitCode=0
Nov 04 01:14:35.956372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:35.956260    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq" event={"ID":"e3be2608-3dbe-4276-b598-7e3f21d0e420","Type":"ContainerDied","Data":"a0045628e60594bfec193dbc679c4cae2573c977a30557b67f45f94499c90adc"}
Nov 04 01:14:36.128245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.118835    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:36.330589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.330559    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:36.438009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.437550    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca\") pod \"0abda11a-e17f-421d-85fe-c40fda1026cc\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") "
Nov 04 01:14:36.438009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.437616    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca\") pod \"0abda11a-e17f-421d-85fe-c40fda1026cc\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") "
Nov 04 01:14:36.438009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.437857    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw\" (UniqueName: \"kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw\") pod \"0abda11a-e17f-421d-85fe-c40fda1026cc\" (UID: \"0abda11a-e17f-421d-85fe-c40fda1026cc\") "
Nov 04 01:14:36.441218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.441188    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca" pod "0abda11a-e17f-421d-85fe-c40fda1026cc" (UID: "0abda11a-e17f-421d-85fe-c40fda1026cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:36.444049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.443968    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw" pod "0abda11a-e17f-421d-85fe-c40fda1026cc" (UID: "0abda11a-e17f-421d-85fe-c40fda1026cc"). InnerVolumeSpecName "kube-api-access-45qzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:36.445904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.445866    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca" pod "0abda11a-e17f-421d-85fe-c40fda1026cc" (UID: "0abda11a-e17f-421d-85fe-c40fda1026cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:36.570120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.570028    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:36.570384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.570364    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abda11a-e17f-421d-85fe-c40fda1026cc-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:36.570511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.570496    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qzw\" (UniqueName: \"kubernetes.io/projected/0abda11a-e17f-421d-85fe-c40fda1026cc-kube-api-access-45qzw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:36.939934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.939897    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:36.966381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.966342    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"
Nov 04 01:14:36.966381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.966344    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl" event={"ID":"0abda11a-e17f-421d-85fe-c40fda1026cc","Type":"ContainerDied","Data":"1638e88ba9849b4d0c8c71ef2af26e6b94635aa01242b5625d04213998261d8c"}
Nov 04 01:14:36.966805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:36.966424    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1638e88ba9849b4d0c8c71ef2af26e6b94635aa01242b5625d04213998261d8c"
Nov 04 01:14:37.108668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.108644    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:37.176379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.176349    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca\") pod \"e3be2608-3dbe-4276-b598-7e3f21d0e420\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") "
Nov 04 01:14:37.176574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.176461    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p\" (UniqueName: \"kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p\") pod \"e3be2608-3dbe-4276-b598-7e3f21d0e420\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") "
Nov 04 01:14:37.176574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.176526    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca\") pod \"e3be2608-3dbe-4276-b598-7e3f21d0e420\" (UID: \"e3be2608-3dbe-4276-b598-7e3f21d0e420\") "
Nov 04 01:14:37.176893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.176870    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca" pod "e3be2608-3dbe-4276-b598-7e3f21d0e420" (UID: "e3be2608-3dbe-4276-b598-7e3f21d0e420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:37.176982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.176968    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:37.178510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.178488    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca" pod "e3be2608-3dbe-4276-b598-7e3f21d0e420" (UID: "e3be2608-3dbe-4276-b598-7e3f21d0e420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:37.178966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.178952    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p" pod "e3be2608-3dbe-4276-b598-7e3f21d0e420" (UID: "e3be2608-3dbe-4276-b598-7e3f21d0e420"). InnerVolumeSpecName "kube-api-access-ljj5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:37.277560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.277530    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3be2608-3dbe-4276-b598-7e3f21d0e420-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:37.277560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.277560    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljj5p\" (UniqueName: \"kubernetes.io/projected/e3be2608-3dbe-4276-b598-7e3f21d0e420-kube-api-access-ljj5p\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:37.770900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.770859    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"]
Nov 04 01:14:37.771635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771594    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e3be2608-3dbe-4276-b598-7e3f21d0e420" containerName="revoke-keys"
Nov 04 01:14:37.771635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771614    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3be2608-3dbe-4276-b598-7e3f21d0e420" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771654    5797 cpu_manager.go:414] "containerMap: removing container" podUID="15154b60-bc67-4d99-b399-ae2dbb675f2c" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771662    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15154b60-bc67-4d99-b399-ae2dbb675f2c" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771678    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0abda11a-e17f-421d-85fe-c40fda1026cc" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771686    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0abda11a-e17f-421d-85fe-c40fda1026cc" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771813    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="15154b60-bc67-4d99-b399-ae2dbb675f2c" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771826    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0abda11a-e17f-421d-85fe-c40fda1026cc" containerName="revoke-keys"
Nov 04 01:14:37.771854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.771842    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3be2608-3dbe-4276-b598-7e3f21d0e420" containerName="revoke-keys"
Nov 04 01:14:37.775548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.775518    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.787814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.787782    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:37.795453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.795417    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"]
Nov 04 01:14:37.883449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.883414    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.883615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.883593    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.883670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.883636    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zk6m\" (UniqueName: \"kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.972865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.972837    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"
Nov 04 01:14:37.973268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.972866    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq" event={"ID":"e3be2608-3dbe-4276-b598-7e3f21d0e420","Type":"ContainerDied","Data":"228a4a3c7cf34e59a60a6c63ed0c7dd6e87fa58220c1d8487e93f1a6e9b31a23"}
Nov 04 01:14:37.973268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.972907    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228a4a3c7cf34e59a60a6c63ed0c7dd6e87fa58220c1d8487e93f1a6e9b31a23"
Nov 04 01:14:37.985109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.985087    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.985230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.985118    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zk6m\" (UniqueName: \"kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.985230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.985206    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.985678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.985658    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:37.987401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:37.987380    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:38.002275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:38.002253    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zk6m\" (UniqueName: \"kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m\") pod \"maas-api-revoke-sub-b7d10b23-2zk75\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:38.093761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:38.093683    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:38.369475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:38.369448    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"]
Nov 04 01:14:38.371380 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:38.371351    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62acd0eb_2fa1_4f0e_8a1e_77653b3fd0cd.slice/crio-cc076fb740c5526b88048fc5e906503caa93153c351df12ad47ee9addc1b79a8 WatchSource:0}: Error finding container cc076fb740c5526b88048fc5e906503caa93153c351df12ad47ee9addc1b79a8: Status 404 returned error can't find the container with id cc076fb740c5526b88048fc5e906503caa93153c351df12ad47ee9addc1b79a8
Nov 04 01:14:38.540130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:38.540085    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:38.980339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:38.980256    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75" event={"ID":"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd","Type":"ContainerStarted","Data":"ebab855f476643304fd7ebeb558edb7cb0cd7130a8304a11182de0e376184bdf"}
Nov 04 01:14:38.980339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:38.980321    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75" event={"ID":"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd","Type":"ContainerStarted","Data":"cc076fb740c5526b88048fc5e906503caa93153c351df12ad47ee9addc1b79a8"}
Nov 04 01:14:39.005717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:39.005672    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75" podStartSLOduration=2.005658353 podStartE2EDuration="2.005658353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:39.001920465 +0000 UTC m=+666.643826739" watchObservedRunningTime="2026-11-04 01:14:39.005658353 +0000 UTC m=+666.647564604"
Nov 04 01:14:39.264616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:39.264587    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:39.992049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:39.991072    5797 generic.go:356] "Generic (PLEG): container finished" podID="62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" containerID="ebab855f476643304fd7ebeb558edb7cb0cd7130a8304a11182de0e376184bdf" exitCode=0
Nov 04 01:14:39.992049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:39.991148    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75" event={"ID":"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd","Type":"ContainerDied","Data":"ebab855f476643304fd7ebeb558edb7cb0cd7130a8304a11182de0e376184bdf"}
Nov 04 01:14:40.024420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.024380    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:40.172404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.172373    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"]
Nov 04 01:14:40.176477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.176460    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.184931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.184899    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"]
Nov 04 01:14:40.211234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.211207    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.211368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.211340    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.211537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.211513    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkb9\" (UniqueName: \"kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.313168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.313082    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.313332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.313166    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.313332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.313255    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkb9\" (UniqueName: \"kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.313925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.313902    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.315355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.315332    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.325279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.325260    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkb9\" (UniqueName: \"kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9\") pod \"maas-api-revoke-sub-102ed2f5-csk9r\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.491471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.491436    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:40.750153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.750088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:40.769380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.769356    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"]
Nov 04 01:14:40.770864 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:40.770831    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be6cc25_4163_4669_935b_9a7b276d1bf0.slice/crio-d89264d35df9152903ca37388598e59da0280a95954a5babf57f609a91bd9f4c WatchSource:0}: Error finding container d89264d35df9152903ca37388598e59da0280a95954a5babf57f609a91bd9f4c: Status 404 returned error can't find the container with id d89264d35df9152903ca37388598e59da0280a95954a5babf57f609a91bd9f4c
Nov 04 01:14:40.998057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:40.998016    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r" event={"ID":"0be6cc25-4163-4669-935b-9a7b276d1bf0","Type":"ContainerStarted","Data":"d89264d35df9152903ca37388598e59da0280a95954a5babf57f609a91bd9f4c"}
Nov 04 01:14:41.133047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.133020    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:41.223543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.223509    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca\") pod \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") "
Nov 04 01:14:41.223733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.223647    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca\") pod \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") "
Nov 04 01:14:41.223733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.223674    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m\" (UniqueName: \"kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m\") pod \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\" (UID: \"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd\") "
Nov 04 01:14:41.223938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.223910    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca" pod "62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" (UID: "62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:41.225733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.225712    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca" pod "62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" (UID: "62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:41.225792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.225775    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m" pod "62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" (UID: "62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd"). InnerVolumeSpecName "kube-api-access-8zk6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:41.325296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.325261    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zk6m\" (UniqueName: \"kubernetes.io/projected/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-kube-api-access-8zk6m\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:41.325296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.325299    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:41.325556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.325315    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:41.577231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:41.577152    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:42.005388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:42.005359    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"
Nov 04 01:14:42.005820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:42.005391    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75" event={"ID":"62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd","Type":"ContainerDied","Data":"cc076fb740c5526b88048fc5e906503caa93153c351df12ad47ee9addc1b79a8"}
Nov 04 01:14:42.005820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:42.005443    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc076fb740c5526b88048fc5e906503caa93153c351df12ad47ee9addc1b79a8"
Nov 04 01:14:42.007082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:42.007051    5797 generic.go:356] "Generic (PLEG): container finished" podID="0be6cc25-4163-4669-935b-9a7b276d1bf0" containerID="fac79af8d9e6aa0d71749378b3c9c41a86dff7ea603a2fd66c2545f471a966a6" exitCode=0
Nov 04 01:14:42.007220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:42.007120    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r" event={"ID":"0be6cc25-4163-4669-935b-9a7b276d1bf0","Type":"ContainerDied","Data":"fac79af8d9e6aa0d71749378b3c9c41a86dff7ea603a2fd66c2545f471a966a6"}
Nov 04 01:14:42.579521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:42.579488    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:43.342704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.342676    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:43.464271 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.464240    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca\") pod \"0be6cc25-4163-4669-935b-9a7b276d1bf0\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") "
Nov 04 01:14:43.464458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.464284    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca\") pod \"0be6cc25-4163-4669-935b-9a7b276d1bf0\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") "
Nov 04 01:14:43.464458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.464389    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9\" (UniqueName: \"kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9\") pod \"0be6cc25-4163-4669-935b-9a7b276d1bf0\" (UID: \"0be6cc25-4163-4669-935b-9a7b276d1bf0\") "
Nov 04 01:14:43.464611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.464584    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca" pod "0be6cc25-4163-4669-935b-9a7b276d1bf0" (UID: "0be6cc25-4163-4669-935b-9a7b276d1bf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:43.464904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.464884    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:43.466476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.466450    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca" pod "0be6cc25-4163-4669-935b-9a7b276d1bf0" (UID: "0be6cc25-4163-4669-935b-9a7b276d1bf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:43.466603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.466562    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9" pod "0be6cc25-4163-4669-935b-9a7b276d1bf0" (UID: "0be6cc25-4163-4669-935b-9a7b276d1bf0"). InnerVolumeSpecName "kube-api-access-prkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:43.504461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.504387    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:43.566044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.565958    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be6cc25-4163-4669-935b-9a7b276d1bf0-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:43.566044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:43.566003    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkb9\" (UniqueName: \"kubernetes.io/projected/0be6cc25-4163-4669-935b-9a7b276d1bf0-kube-api-access-prkb9\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:44.057373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:44.057341    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"
Nov 04 01:14:44.057373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:44.057367    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r" event={"ID":"0be6cc25-4163-4669-935b-9a7b276d1bf0","Type":"ContainerDied","Data":"d89264d35df9152903ca37388598e59da0280a95954a5babf57f609a91bd9f4c"}
Nov 04 01:14:44.057655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:44.057408    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89264d35df9152903ca37388598e59da0280a95954a5babf57f609a91bd9f4c"
Nov 04 01:14:44.300730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:44.300692    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:44.991940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:44.991900    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:45.882269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:45.882235    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:46.596882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:46.596845    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:46.811186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:46.811155    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:47.381283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:47.381245    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:48.101115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.101083    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"]
Nov 04 01:14:48.101846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.101829    5797 cpu_manager.go:414] "containerMap: removing container" podUID="62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" containerName="revoke-keys"
Nov 04 01:14:48.101905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.101849    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" containerName="revoke-keys"
Nov 04 01:14:48.101905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.101885    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0be6cc25-4163-4669-935b-9a7b276d1bf0" containerName="revoke-keys"
Nov 04 01:14:48.101905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.101895    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be6cc25-4163-4669-935b-9a7b276d1bf0" containerName="revoke-keys"
Nov 04 01:14:48.102057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.102047    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" containerName="revoke-keys"
Nov 04 01:14:48.102095 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.102066    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be6cc25-4163-4669-935b-9a7b276d1bf0" containerName="revoke-keys"
Nov 04 01:14:48.106082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.106066    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.110577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.110555    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:14:48.110801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.110781    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:14:48.113395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.113331    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"]
Nov 04 01:14:48.119743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.119709    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqgg2\" (UniqueName: \"kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.119867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.119823    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.119937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.119892    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.221483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.221447    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqgg2\" (UniqueName: \"kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.221682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.221515    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.221682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.221554    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.222231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.222200    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.223911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.223888    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.236639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.236612    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqgg2\" (UniqueName: \"kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2\") pod \"maas-api-revoke-sub-faf5823d-hm8vn\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:48.288188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.288145    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:48.472094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:48.465824    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:49.300414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:49.294814    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"]
Nov 04 01:14:49.702690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:49.702606    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:50.337071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:50.336429    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn" event={"ID":"52c45138-01f4-4226-a249-0b786061a5ac","Type":"ContainerStarted","Data":"f25397224eea6d11180ccf1eacd0d7c0102c8d9630d344b9a4453e95af40ae55"}
Nov 04 01:14:50.337071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:50.336503    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn" event={"ID":"52c45138-01f4-4226-a249-0b786061a5ac","Type":"ContainerStarted","Data":"c4918bbb4a5d5d232d96eb86094add2fe5f22cf55e91da1eaf576149e0974b78"}
Nov 04 01:14:50.399013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:50.398162    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn" podStartSLOduration=2.39814137 podStartE2EDuration="2.39814137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:50.375638359 +0000 UTC m=+678.017544718" watchObservedRunningTime="2026-11-04 01:14:50.39814137 +0000 UTC m=+678.040047623"
Nov 04 01:14:50.979011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:50.978528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:51.316011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:51.315777    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:14:51.377583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:51.377529    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:14:51.418560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:51.418524    5797 generic.go:356] "Generic (PLEG): container finished" podID="52c45138-01f4-4226-a249-0b786061a5ac" containerID="f25397224eea6d11180ccf1eacd0d7c0102c8d9630d344b9a4453e95af40ae55" exitCode=0
Nov 04 01:14:51.419006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:51.418958    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn" event={"ID":"52c45138-01f4-4226-a249-0b786061a5ac","Type":"ContainerDied","Data":"f25397224eea6d11180ccf1eacd0d7c0102c8d9630d344b9a4453e95af40ae55"}
Nov 04 01:14:52.237128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.237078    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:14:52.283191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.282073    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:14:52.381112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.381067    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:14:52.382699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.382675    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:14:52.603686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.603657    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:14:52.848528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.848184    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:52.870088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.870053    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:52.870773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.870743    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2\" (UniqueName: \"kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2\") pod \"52c45138-01f4-4226-a249-0b786061a5ac\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") "
Nov 04 01:14:52.870900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.870797    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca\") pod \"52c45138-01f4-4226-a249-0b786061a5ac\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") "
Nov 04 01:14:52.870960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.870931    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca\") pod \"52c45138-01f4-4226-a249-0b786061a5ac\" (UID: \"52c45138-01f4-4226-a249-0b786061a5ac\") "
Nov 04 01:14:52.874283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.874253    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca" pod "52c45138-01f4-4226-a249-0b786061a5ac" (UID: "52c45138-01f4-4226-a249-0b786061a5ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:52.877567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.877487    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca" pod "52c45138-01f4-4226-a249-0b786061a5ac" (UID: "52c45138-01f4-4226-a249-0b786061a5ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:52.878860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.878826    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2" pod "52c45138-01f4-4226-a249-0b786061a5ac" (UID: "52c45138-01f4-4226-a249-0b786061a5ac"). InnerVolumeSpecName "kube-api-access-vqgg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:52.972483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.972449    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqgg2\" (UniqueName: \"kubernetes.io/projected/52c45138-01f4-4226-a249-0b786061a5ac-kube-api-access-vqgg2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:52.972483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.972482    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:52.972686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:52.972496    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c45138-01f4-4226-a249-0b786061a5ac-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:53.554969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:53.554935    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"
Nov 04 01:14:53.554969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:53.554964    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn" event={"ID":"52c45138-01f4-4226-a249-0b786061a5ac","Type":"ContainerDied","Data":"c4918bbb4a5d5d232d96eb86094add2fe5f22cf55e91da1eaf576149e0974b78"}
Nov 04 01:14:53.555446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:53.555045    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4918bbb4a5d5d232d96eb86094add2fe5f22cf55e91da1eaf576149e0974b78"
Nov 04 01:14:53.594536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:53.594505    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:54.204029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.201162    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"]
Nov 04 01:14:54.204029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.202579    5797 cpu_manager.go:414] "containerMap: removing container" podUID="52c45138-01f4-4226-a249-0b786061a5ac" containerName="revoke-keys"
Nov 04 01:14:54.204029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.202600    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c45138-01f4-4226-a249-0b786061a5ac" containerName="revoke-keys"
Nov 04 01:14:54.204029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.202828    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c45138-01f4-4226-a249-0b786061a5ac" containerName="revoke-keys"
Nov 04 01:14:54.211242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.211214    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.213921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.213894    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"]
Nov 04 01:14:54.215735 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.215712    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:14:54.217106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.217063    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:14:54.288502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.288467    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfx4p\" (UniqueName: \"kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.288664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.288566    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.288664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.288606    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.390129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.390096    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfx4p\" (UniqueName: \"kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.390291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.390221    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.390291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.390273    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.390977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.390936    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.392922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.392896    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.404439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.404403    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfx4p\" (UniqueName: \"kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p\") pod \"maas-api-revoke-sub-eafca183-q4zrz\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.405136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.405116    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"]
Nov 04 01:14:54.409730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.409711    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.416538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.416517    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"]
Nov 04 01:14:54.491918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.491832    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.492065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.492021    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.492198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.492181    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pls6\" (UniqueName: \"kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.520572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.520545    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:54.531455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.531431    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:54.593819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.593783    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pls6\" (UniqueName: \"kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.594249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.593872    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.594249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.593975    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.594819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.594797    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.600458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.600388    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.611375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.611347    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pls6\" (UniqueName: \"kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6\") pod \"maas-api-revoke-sub-81248b03-kl86r\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.736080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.736047    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:54.844020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:54.843975    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"]
Nov 04 01:14:54.851558 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:54.851530    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5efa76_ad37_4da3_8c5c_9f2e5807f5c9.slice/crio-3187cd19c08fbedf83d274422f9c382c2b010a6984748c2615489fea50fa879f WatchSource:0}: Error finding container 3187cd19c08fbedf83d274422f9c382c2b010a6984748c2615489fea50fa879f: Status 404 returned error can't find the container with id 3187cd19c08fbedf83d274422f9c382c2b010a6984748c2615489fea50fa879f
Nov 04 01:14:55.017305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.017271    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"]
Nov 04 01:14:55.018764 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:55.018738    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd941a2d_1526_44f4_8648_3523597644f5.slice/crio-a2c617d47ca397ca8ac9a417f0207d8fd70e3e304ddc99fa919e5b6f074744dd WatchSource:0}: Error finding container a2c617d47ca397ca8ac9a417f0207d8fd70e3e304ddc99fa919e5b6f074744dd: Status 404 returned error can't find the container with id a2c617d47ca397ca8ac9a417f0207d8fd70e3e304ddc99fa919e5b6f074744dd
Nov 04 01:14:55.410077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.410045    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:55.574766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.574685    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz" event={"ID":"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9","Type":"ContainerStarted","Data":"cb4b32f76e1d731f3e077e528dab64d20182c3a55aa51d6b92180f8184bbfc34"}
Nov 04 01:14:55.574766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.574747    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz" event={"ID":"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9","Type":"ContainerStarted","Data":"3187cd19c08fbedf83d274422f9c382c2b010a6984748c2615489fea50fa879f"}
Nov 04 01:14:55.576317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.576288    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r" event={"ID":"cd941a2d-1526-44f4-8648-3523597644f5","Type":"ContainerStarted","Data":"132beb9ad93bf941bb8da31ee1d2957bf32addc0d35848ba0029ee4261d0aab4"}
Nov 04 01:14:55.576519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.576338    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r" event={"ID":"cd941a2d-1526-44f4-8648-3523597644f5","Type":"ContainerStarted","Data":"a2c617d47ca397ca8ac9a417f0207d8fd70e3e304ddc99fa919e5b6f074744dd"}
Nov 04 01:14:55.599912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.599852    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz" podStartSLOduration=1.599834167 podStartE2EDuration="1.599834167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:55.597038532 +0000 UTC m=+683.238944815" watchObservedRunningTime="2026-11-04 01:14:55.599834167 +0000 UTC m=+683.241740420"
Nov 04 01:14:55.606493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.606467    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"]
Nov 04 01:14:55.610592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.610573    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.625978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.625923    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r" podStartSLOduration=1.625903144 podStartE2EDuration="1.625903144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:55.620515414 +0000 UTC m=+683.262421664" watchObservedRunningTime="2026-11-04 01:14:55.625903144 +0000 UTC m=+683.267809396"
Nov 04 01:14:55.634034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.630096    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"]
Nov 04 01:14:55.712825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.712793    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.712971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.712904    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.713082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.713066    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntxf\" (UniqueName: \"kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.814644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.814610    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntxf\" (UniqueName: \"kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.814810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.814677    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.814810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.814724    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.815416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.815388    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.817180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.817152    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.827153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.827094    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntxf\" (UniqueName: \"kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf\") pod \"maas-api-revoke-sub-23db2f58-5vflz\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:55.955653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:55.955620    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:56.108395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.108316    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:56.237642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.237612    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"]
Nov 04 01:14:56.238683 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:56.238650    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae8f72b_2344_412e_86b4_a7eb1080569d.slice/crio-813dc74a055ae5aa378360cde6e23986204062725b570b62eb6e8c589802e512 WatchSource:0}: Error finding container 813dc74a055ae5aa378360cde6e23986204062725b570b62eb6e8c589802e512: Status 404 returned error can't find the container with id 813dc74a055ae5aa378360cde6e23986204062725b570b62eb6e8c589802e512
Nov 04 01:14:56.255351 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:14:56.255318    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:14:56.584862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.584831    5797 generic.go:356] "Generic (PLEG): container finished" podID="cd941a2d-1526-44f4-8648-3523597644f5" containerID="132beb9ad93bf941bb8da31ee1d2957bf32addc0d35848ba0029ee4261d0aab4" exitCode=0
Nov 04 01:14:56.585020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.584881    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r" event={"ID":"cd941a2d-1526-44f4-8648-3523597644f5","Type":"ContainerDied","Data":"132beb9ad93bf941bb8da31ee1d2957bf32addc0d35848ba0029ee4261d0aab4"}
Nov 04 01:14:56.586857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.586824    5797 generic.go:356] "Generic (PLEG): container finished" podID="fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" containerID="cb4b32f76e1d731f3e077e528dab64d20182c3a55aa51d6b92180f8184bbfc34" exitCode=0
Nov 04 01:14:56.587020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.586857    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz" event={"ID":"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9","Type":"ContainerDied","Data":"cb4b32f76e1d731f3e077e528dab64d20182c3a55aa51d6b92180f8184bbfc34"}
Nov 04 01:14:56.588638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.588614    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz" event={"ID":"5ae8f72b-2344-412e-86b4-a7eb1080569d","Type":"ContainerStarted","Data":"fbc0a0995a81f88e3d8390fad1b76d0eef44eb174772dcb67f39f0974526af01"}
Nov 04 01:14:56.588714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.588660    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz" event={"ID":"5ae8f72b-2344-412e-86b4-a7eb1080569d","Type":"ContainerStarted","Data":"813dc74a055ae5aa378360cde6e23986204062725b570b62eb6e8c589802e512"}
Nov 04 01:14:56.644091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.644046    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz" podStartSLOduration=1.644031891 podStartE2EDuration="1.644031891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:56.642971015 +0000 UTC m=+684.284877266" watchObservedRunningTime="2026-11-04 01:14:56.644031891 +0000 UTC m=+684.285938141"
Nov 04 01:14:56.803371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:56.803291    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:57.489148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.489118    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:57.597840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.597805    5797 generic.go:356] "Generic (PLEG): container finished" podID="5ae8f72b-2344-412e-86b4-a7eb1080569d" containerID="fbc0a0995a81f88e3d8390fad1b76d0eef44eb174772dcb67f39f0974526af01" exitCode=0
Nov 04 01:14:57.598016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.597854    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz" event={"ID":"5ae8f72b-2344-412e-86b4-a7eb1080569d","Type":"ContainerDied","Data":"fbc0a0995a81f88e3d8390fad1b76d0eef44eb174772dcb67f39f0974526af01"}
Nov 04 01:14:57.789248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.789225    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:57.815217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.815187    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.838157    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca\") pod \"cd941a2d-1526-44f4-8648-3523597644f5\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") "
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.838212    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6\" (UniqueName: \"kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6\") pod \"cd941a2d-1526-44f4-8648-3523597644f5\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") "
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.838462    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca\") pod \"cd941a2d-1526-44f4-8648-3523597644f5\" (UID: \"cd941a2d-1526-44f4-8648-3523597644f5\") "
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.838517    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca" pod "cd941a2d-1526-44f4-8648-3523597644f5" (UID: "cd941a2d-1526-44f4-8648-3523597644f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.838951    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.844522    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca" pod "cd941a2d-1526-44f4-8648-3523597644f5" (UID: "cd941a2d-1526-44f4-8648-3523597644f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:57.845361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.845087    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6" pod "cd941a2d-1526-44f4-8648-3523597644f5" (UID: "cd941a2d-1526-44f4-8648-3523597644f5"). InnerVolumeSpecName "kube-api-access-2pls6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:57.939618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.939590    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p\" (UniqueName: \"kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p\") pod \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") "
Nov 04 01:14:57.939772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.939666    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca\") pod \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") "
Nov 04 01:14:57.939772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.939709    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca\") pod \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\" (UID: \"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9\") "
Nov 04 01:14:57.940174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.940148    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd941a2d-1526-44f4-8648-3523597644f5-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:57.940315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.940184    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pls6\" (UniqueName: \"kubernetes.io/projected/cd941a2d-1526-44f4-8648-3523597644f5-kube-api-access-2pls6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:57.940315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.940152    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca" pod "fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" (UID: "fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:57.941621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.941595    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca" pod "fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" (UID: "fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:57.941755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.941734    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p" pod "fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" (UID: "fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9"). InnerVolumeSpecName "kube-api-access-dfx4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:57.957498 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.957474    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"]
Nov 04 01:14:57.957984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.957971    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cd941a2d-1526-44f4-8648-3523597644f5" containerName="revoke-keys"
Nov 04 01:14:57.958047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.957985    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd941a2d-1526-44f4-8648-3523597644f5" containerName="revoke-keys"
Nov 04 01:14:57.958047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.958021    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" containerName="revoke-keys"
Nov 04 01:14:57.958047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.958026    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" containerName="revoke-keys"
Nov 04 01:14:57.958133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.958128    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd941a2d-1526-44f4-8648-3523597644f5" containerName="revoke-keys"
Nov 04 01:14:57.958167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.958138    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" containerName="revoke-keys"
Nov 04 01:14:57.961660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.961646    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:57.971210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:57.971187    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"]
Nov 04 01:14:58.041482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.041406    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzvj\" (UniqueName: \"kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.041482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.041445    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.041651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.041593    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.041726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.041711    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:58.041759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.041732    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:58.041759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.041743    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfx4p\" (UniqueName: \"kubernetes.io/projected/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9-kube-api-access-dfx4p\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:58.143206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.143171    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.143362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.143263    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzvj\" (UniqueName: \"kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.143362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.143286    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.143806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.143784    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.145548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.145525    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.155900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.155878    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzvj\" (UniqueName: \"kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj\") pod \"maas-api-revoke-sub-3ad1e043-hllkj\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.276506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.276467    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:14:58.425272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.425239    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:58.552791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.552766    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"]
Nov 04 01:14:58.553848 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:14:58.553821    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc917cb96_fd72_4b11_9c10_2acad6284171.slice/crio-88ad24df0ddce587ae294b22fe2b786bbf2a93410389b7363e40f532e6b3252a WatchSource:0}: Error finding container 88ad24df0ddce587ae294b22fe2b786bbf2a93410389b7363e40f532e6b3252a: Status 404 returned error can't find the container with id 88ad24df0ddce587ae294b22fe2b786bbf2a93410389b7363e40f532e6b3252a
Nov 04 01:14:58.613726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.613692    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj" event={"ID":"c917cb96-fd72-4b11-9c10-2acad6284171","Type":"ContainerStarted","Data":"88ad24df0ddce587ae294b22fe2b786bbf2a93410389b7363e40f532e6b3252a"}
Nov 04 01:14:58.615039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.615016    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"
Nov 04 01:14:58.615145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.615031    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz" event={"ID":"fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9","Type":"ContainerDied","Data":"3187cd19c08fbedf83d274422f9c382c2b010a6984748c2615489fea50fa879f"}
Nov 04 01:14:58.615145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.615092    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3187cd19c08fbedf83d274422f9c382c2b010a6984748c2615489fea50fa879f"
Nov 04 01:14:58.616278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.616256    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"
Nov 04 01:14:58.616368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.616278    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r" event={"ID":"cd941a2d-1526-44f4-8648-3523597644f5","Type":"ContainerDied","Data":"a2c617d47ca397ca8ac9a417f0207d8fd70e3e304ddc99fa919e5b6f074744dd"}
Nov 04 01:14:58.616368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.616312    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2c617d47ca397ca8ac9a417f0207d8fd70e3e304ddc99fa919e5b6f074744dd"
Nov 04 01:14:58.748063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.747980    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:14:58.851264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.851233    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca\") pod \"5ae8f72b-2344-412e-86b4-a7eb1080569d\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") "
Nov 04 01:14:58.851664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.851396    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca\") pod \"5ae8f72b-2344-412e-86b4-a7eb1080569d\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") "
Nov 04 01:14:58.851664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.851440    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf\" (UniqueName: \"kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf\") pod \"5ae8f72b-2344-412e-86b4-a7eb1080569d\" (UID: \"5ae8f72b-2344-412e-86b4-a7eb1080569d\") "
Nov 04 01:14:58.851831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.851804    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca" pod "5ae8f72b-2344-412e-86b4-a7eb1080569d" (UID: "5ae8f72b-2344-412e-86b4-a7eb1080569d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:14:58.853241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.853218    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca" pod "5ae8f72b-2344-412e-86b4-a7eb1080569d" (UID: "5ae8f72b-2344-412e-86b4-a7eb1080569d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:14:58.853576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.853557    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf" pod "5ae8f72b-2344-412e-86b4-a7eb1080569d" (UID: "5ae8f72b-2344-412e-86b4-a7eb1080569d"). InnerVolumeSpecName "kube-api-access-cntxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:14:58.953653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.953619    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:58.953653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.953646    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntxf\" (UniqueName: \"kubernetes.io/projected/5ae8f72b-2344-412e-86b4-a7eb1080569d-kube-api-access-cntxf\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:58.953653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:58.953656    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae8f72b-2344-412e-86b4-a7eb1080569d-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:14:59.298742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:59.298711    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:14:59.624443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:59.624409    5797 generic.go:356] "Generic (PLEG): container finished" podID="c917cb96-fd72-4b11-9c10-2acad6284171" containerID="6cb8b0c1ba9f6bf2139b8d76618e109234c847976eb820b3e7a12b652ba8e13b" exitCode=0
Nov 04 01:14:59.624599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:59.624496    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj" event={"ID":"c917cb96-fd72-4b11-9c10-2acad6284171","Type":"ContainerDied","Data":"6cb8b0c1ba9f6bf2139b8d76618e109234c847976eb820b3e7a12b652ba8e13b"}
Nov 04 01:14:59.626001 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:59.625965    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz" event={"ID":"5ae8f72b-2344-412e-86b4-a7eb1080569d","Type":"ContainerDied","Data":"813dc74a055ae5aa378360cde6e23986204062725b570b62eb6e8c589802e512"}
Nov 04 01:14:59.626135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:59.626012    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="813dc74a055ae5aa378360cde6e23986204062725b570b62eb6e8c589802e512"
Nov 04 01:14:59.626135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:14:59.626015    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"
Nov 04 01:15:00.072282 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.072198    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:00.145284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.145250    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"]
Nov 04 01:15:00.146104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.146072    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5ae8f72b-2344-412e-86b4-a7eb1080569d" containerName="revoke-keys"
Nov 04 01:15:00.146104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.146103    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae8f72b-2344-412e-86b4-a7eb1080569d" containerName="revoke-keys"
Nov 04 01:15:00.146320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.146259    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae8f72b-2344-412e-86b4-a7eb1080569d" containerName="revoke-keys"
Nov 04 01:15:00.150269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.150244    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"]
Nov 04 01:15:00.150408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.150376    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.154024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.154002    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.154151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.154109    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"]
Nov 04 01:15:00.161945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.161315    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"]
Nov 04 01:15:00.161945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.161849    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.165686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.165667    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"]
Nov 04 01:15:00.165930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.165905    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.169703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.169685    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"]
Nov 04 01:15:00.198076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.198047    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"]
Nov 04 01:15:00.204743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.204720    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"]
Nov 04 01:15:00.211005 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.210968    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"]
Nov 04 01:15:00.216033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.215899    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"]
Nov 04 01:15:00.216155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.216046    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.298515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298478    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.298772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298529    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfkc\" (UniqueName: \"kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc\") pod \"maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.298772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298574    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.298772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298614    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.298772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298656    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdshh\" (UniqueName: \"kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.298772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298703    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.307775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298788    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwp7\" (UniqueName: \"kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.307775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298841    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.307775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298889    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjhh\" (UniqueName: \"kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh\") pod \"maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.307775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.298945    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7xz\" (UniqueName: \"kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz\") pod \"maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.400104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400069    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.400104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400103    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfkc\" (UniqueName: \"kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc\") pod \"maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.400325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400133    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.400325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400163    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.400325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400193    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdshh\" (UniqueName: \"kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.400325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400230    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.400325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400310    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwp7\" (UniqueName: \"kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.400610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400360    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.400610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400405    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjhh\" (UniqueName: \"kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh\") pod \"maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.400610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400461    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7xz\" (UniqueName: \"kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz\") pod \"maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.400930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400901    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.401070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.400974    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.401156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.401064    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.401156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.401096    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.401542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.401519    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.412910 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.412876    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwp7\" (UniqueName: \"kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7\") pod \"maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.413169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.413143    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfkc\" (UniqueName: \"kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc\") pod \"maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.413252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.413163    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjhh\" (UniqueName: \"kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh\") pod \"maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.413357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.413335    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdshh\" (UniqueName: \"kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.413410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.413393    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7xz\" (UniqueName: \"kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz\") pod \"maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.482579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.478426    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:00.500226 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.500198    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:00.528250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.528215    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:00.538956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.538920    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:00.549058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.547334    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:00.968104 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:00.968033    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b61f6_bb9f_4051_a743_bebeed856a7e.slice/crio-d689dcd3741d940cd9753f1d2aa73faf10f75b7fe4a62008c40cfa2b66500e11 WatchSource:0}: Error finding container d689dcd3741d940cd9753f1d2aa73faf10f75b7fe4a62008c40cfa2b66500e11: Status 404 returned error can't find the container with id d689dcd3741d940cd9753f1d2aa73faf10f75b7fe4a62008c40cfa2b66500e11
Nov 04 01:15:00.972626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.972577    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"]
Nov 04 01:15:00.979425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:00.979401    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"]
Nov 04 01:15:01.037298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.036940    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:15:01.083980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.083950    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"]
Nov 04 01:15:01.128537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.128303    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"]
Nov 04 01:15:01.170964 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.170434    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:01.173043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.172897    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca\") pod \"c917cb96-fd72-4b11-9c10-2acad6284171\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") "
Nov 04 01:15:01.174086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.173350    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj\" (UniqueName: \"kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj\") pod \"c917cb96-fd72-4b11-9c10-2acad6284171\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") "
Nov 04 01:15:01.174086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.173444    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca\") pod \"c917cb96-fd72-4b11-9c10-2acad6284171\" (UID: \"c917cb96-fd72-4b11-9c10-2acad6284171\") "
Nov 04 01:15:01.174721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.174403    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca" pod "c917cb96-fd72-4b11-9c10-2acad6284171" (UID: "c917cb96-fd72-4b11-9c10-2acad6284171"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:01.251751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.251649    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca" pod "c917cb96-fd72-4b11-9c10-2acad6284171" (UID: "c917cb96-fd72-4b11-9c10-2acad6284171"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:01.271219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.271142    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj" pod "c917cb96-fd72-4b11-9c10-2acad6284171" (UID: "c917cb96-fd72-4b11-9c10-2acad6284171"). InnerVolumeSpecName "kube-api-access-zqzvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:01.295024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.293453    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:01.295024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.293489    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzvj\" (UniqueName: \"kubernetes.io/projected/c917cb96-fd72-4b11-9c10-2acad6284171-kube-api-access-zqzvj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:01.295024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.293509    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c917cb96-fd72-4b11-9c10-2acad6284171-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:01.446461 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:01.445463    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc48035_c08c_41f2_91eb_7bd56d7deba3.slice/crio-f1ce1259ac6d5009e3bcaeea343988b6a85591df950a00ef5f869e4a0a02af1c WatchSource:0}: Error finding container f1ce1259ac6d5009e3bcaeea343988b6a85591df950a00ef5f869e4a0a02af1c: Status 404 returned error can't find the container with id f1ce1259ac6d5009e3bcaeea343988b6a85591df950a00ef5f869e4a0a02af1c
Nov 04 01:15:01.469319 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:01.457441    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"]
Nov 04 01:15:02.031015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.016624    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerStarted","Data":"d689dcd3741d940cd9753f1d2aa73faf10f75b7fe4a62008c40cfa2b66500e11"}
Nov 04 01:15:02.065133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.064093    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerStarted","Data":"b0cc90ef801c7f0b8e33bfbd837c9520e62ce6470e53fecee84ec7b900a73ce1"}
Nov 04 01:15:02.067311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.067221    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" event={"ID":"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9","Type":"ContainerStarted","Data":"25d3606910f08f90a3f09e3b82d0e7a581608bfb6972f017696a8f07ce3775b4"}
Nov 04 01:15:02.069406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.069382    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj" event={"ID":"c917cb96-fd72-4b11-9c10-2acad6284171","Type":"ContainerDied","Data":"88ad24df0ddce587ae294b22fe2b786bbf2a93410389b7363e40f532e6b3252a"}
Nov 04 01:15:02.069598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.069580    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ad24df0ddce587ae294b22fe2b786bbf2a93410389b7363e40f532e6b3252a"
Nov 04 01:15:02.069831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.069816    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"
Nov 04 01:15:02.079329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.079249    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerStarted","Data":"f1ce1259ac6d5009e3bcaeea343988b6a85591df950a00ef5f869e4a0a02af1c"}
Nov 04 01:15:02.088026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.083859    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerStarted","Data":"8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d"}
Nov 04 01:15:02.088026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.083907    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerStarted","Data":"f08344f7ec0a1a24c7f079ea5bd424c24242681396a9f8714fa56c4034328a1e"}
Nov 04 01:15:02.289185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.273372    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" podStartSLOduration=2.27334918 podStartE2EDuration="2.27334918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:02.096591111 +0000 UTC m=+689.738497386" watchObservedRunningTime="2026-11-04 01:15:02.27334918 +0000 UTC m=+689.915255434"
Nov 04 01:15:02.289185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.278128    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" podStartSLOduration=2.278114925 podStartE2EDuration="2.278114925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:02.131811814 +0000 UTC m=+689.773718066" watchObservedRunningTime="2026-11-04 01:15:02.278114925 +0000 UTC m=+689.920021175"
Nov 04 01:15:02.289185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.285809    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"]
Nov 04 01:15:02.289185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.287414    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c917cb96-fd72-4b11-9c10-2acad6284171" containerName="revoke-keys"
Nov 04 01:15:02.289185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.287435    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c917cb96-fd72-4b11-9c10-2acad6284171" containerName="revoke-keys"
Nov 04 01:15:02.289185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.287573    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c917cb96-fd72-4b11-9c10-2acad6284171" containerName="revoke-keys"
Nov 04 01:15:02.302570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.301073    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"]
Nov 04 01:15:02.302570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.301158    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.437522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.437483    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"]
Nov 04 01:15:02.529907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.529875    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.530126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.530101    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.530385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.530364    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.530523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.530507    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.530645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.530628    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.530816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.530799    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.532797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.530951    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4h2t\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.532797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.531070    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.532797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.531114    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.543350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.543260    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"]
Nov 04 01:15:02.545474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.545451    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.559494 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:02.559422    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b61f6_bb9f_4051_a743_bebeed856a7e.slice/crio-720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7932a793_995c_4f0c_9d4e_51cc775b8b54.slice/crio-8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b61f6_bb9f_4051_a743_bebeed856a7e.slice/crio-conmon-720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7932a793_995c_4f0c_9d4e_51cc775b8b54.slice/crio-conmon-8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 01:15:02.561281 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:02.559974    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b61f6_bb9f_4051_a743_bebeed856a7e.slice/crio-conmon-720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7932a793_995c_4f0c_9d4e_51cc775b8b54.slice/crio-conmon-8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc917cb96_fd72_4b11_9c10_2acad6284171.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7932a793_995c_4f0c_9d4e_51cc775b8b54.slice/crio-8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 01:15:02.571739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.571711    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ba844c-openshift-default-dockercfg-xd85j\""
Nov 04 01:15:02.633446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633407    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633455    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4h2t\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633485    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633521    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633604    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633640    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633757    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633785    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.633865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.633817    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.635705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.635085    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.635705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.635465    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.635705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.635666    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.636393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.636361    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.636504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.636482    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.647142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.647080    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.648498 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.648471    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4h2t\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.657587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.657557    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.675897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.674167    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.735176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735139    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735236    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735301    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735347    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735449    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735482    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735564    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhd8\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735597    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.735655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.735617    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.737153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.737123    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:02.836537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836504    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.836707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836545    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.836707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836581    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.836707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836609    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.836897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836833    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhd8\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.836952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836901    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.836952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836932    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.836983    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.837095    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.837094    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.837279    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.837300    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.837364    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.837704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.837673    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.839096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.839074    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.839331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.839313    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.841208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.841190    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:02.852676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.852626    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhd8\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.856041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.855962    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:02.931695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:02.931659    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:03.119918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.119592    5797 generic.go:356] "Generic (PLEG): container finished" podID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerID="720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8" exitCode=60
Nov 04 01:15:03.119918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.119718    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerDied","Data":"720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8"}
Nov 04 01:15:03.120443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.120188    5797 scope.go:122] "RemoveContainer" containerID="720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8"
Nov 04 01:15:03.124286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.124176    5797 generic.go:356] "Generic (PLEG): container finished" podID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerID="759a5760186c67073f9d77cf0225758cd693586c24117d4c60b762b8c7fefed9" exitCode=60
Nov 04 01:15:03.124286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.124205    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerDied","Data":"759a5760186c67073f9d77cf0225758cd693586c24117d4c60b762b8c7fefed9"}
Nov 04 01:15:03.124443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.124424    5797 scope.go:122] "RemoveContainer" containerID="759a5760186c67073f9d77cf0225758cd693586c24117d4c60b762b8c7fefed9"
Nov 04 01:15:03.128456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.128436    5797 generic.go:356] "Generic (PLEG): container finished" podID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerID="5f40545a069abbc32219f5ac40f4da24557e47259de888aa20ab4ddd7fe21fcb" exitCode=60
Nov 04 01:15:03.128551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.128498    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" event={"ID":"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9","Type":"ContainerDied","Data":"5f40545a069abbc32219f5ac40f4da24557e47259de888aa20ab4ddd7fe21fcb"}
Nov 04 01:15:03.128763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.128749    5797 scope.go:122] "RemoveContainer" containerID="5f40545a069abbc32219f5ac40f4da24557e47259de888aa20ab4ddd7fe21fcb"
Nov 04 01:15:03.131020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.130617    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerStarted","Data":"2716e8448b1f23089dc491886b061befd82565ce773725599c333c461d81cc59"}
Nov 04 01:15:03.136428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.135181    5797 generic.go:356] "Generic (PLEG): container finished" podID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerID="8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d" exitCode=60
Nov 04 01:15:03.136428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.135290    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerDied","Data":"8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d"}
Nov 04 01:15:03.136428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.135585    5797 scope.go:122] "RemoveContainer" containerID="8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d"
Nov 04 01:15:03.215460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.209565    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"]
Nov 04 01:15:03.216844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.216600    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" podStartSLOduration=3.216581318 podStartE2EDuration="3.216581318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:03.201667806 +0000 UTC m=+690.843574050" watchObservedRunningTime="2026-11-04 01:15:03.216581318 +0000 UTC m=+690.858487570"
Nov 04 01:15:03.237844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.237802    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:03.237954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.237899    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:03.237954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.237942    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:03.301180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.301147    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"]
Nov 04 01:15:03.371475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.366425    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:03.371475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.366602    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:03.371475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.366675    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:03.810232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:03.798428    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:04.251264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.242108    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" event={"ID":"98ed529b-8904-4f22-baae-e7635a3090b3","Type":"ContainerStarted","Data":"6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910"}
Nov 04 01:15:04.251264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.242164    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" event={"ID":"98ed529b-8904-4f22-baae-e7635a3090b3","Type":"ContainerStarted","Data":"bc0dcbdd31db8a5bc98769bfa4d3a884054927e995e48fba68371cc0f5ad148e"}
Nov 04 01:15:04.251264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.244745    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerStarted","Data":"42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2"}
Nov 04 01:15:04.270438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.266131    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerStarted","Data":"5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc"}
Nov 04 01:15:04.315594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.305833    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podStartSLOduration=2.305810493 podStartE2EDuration="2.305810493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:04.265650703 +0000 UTC m=+691.907556954" watchObservedRunningTime="2026-11-04 01:15:04.305810493 +0000 UTC m=+691.947716744"
Nov 04 01:15:04.336642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.336610    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" event={"ID":"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9","Type":"ContainerStarted","Data":"e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4"}
Nov 04 01:15:04.342735 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.342708    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerDied","Data":"2716e8448b1f23089dc491886b061befd82565ce773725599c333c461d81cc59"}
Nov 04 01:15:04.343084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.342962    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" podStartSLOduration=4.342945462 podStartE2EDuration="4.342945462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:04.332861889 +0000 UTC m=+691.974768166" watchObservedRunningTime="2026-11-04 01:15:04.342945462 +0000 UTC m=+691.984851715"
Nov 04 01:15:04.343560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.343542    5797 scope.go:122] "RemoveContainer" containerID="2716e8448b1f23089dc491886b061befd82565ce773725599c333c461d81cc59"
Nov 04 01:15:04.346669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.346647    5797 generic.go:356] "Generic (PLEG): container finished" podID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerID="2716e8448b1f23089dc491886b061befd82565ce773725599c333c461d81cc59" exitCode=60
Nov 04 01:15:04.545040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.543868    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" podStartSLOduration=4.543847306 podStartE2EDuration="4.543847306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:04.369921899 +0000 UTC m=+692.011828150" watchObservedRunningTime="2026-11-04 01:15:04.543847306 +0000 UTC m=+692.185753570"
Nov 04 01:15:04.679031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.678795    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerStarted","Data":"9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43"}
Nov 04 01:15:04.679031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.678890    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" event={"ID":"24765e2f-e83c-4abb-92e2-2d04ec59899a","Type":"ContainerStarted","Data":"3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151"}
Nov 04 01:15:04.679031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.678924    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" event={"ID":"24765e2f-e83c-4abb-92e2-2d04ec59899a","Type":"ContainerStarted","Data":"418f35f3dac9908bd770de8a2a93638f1809a10f76a80328d0548475472dc3a9"}
Nov 04 01:15:04.773274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.773211    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podStartSLOduration=2.773188738 podStartE2EDuration="2.773188738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:04.606565238 +0000 UTC m=+692.248471514" watchObservedRunningTime="2026-11-04 01:15:04.773188738 +0000 UTC m=+692.415094989"
Nov 04 01:15:04.845445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.844055    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 04 01:15:04.845445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.844136    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 04 01:15:04.845445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.842420    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:04.939583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.939544    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:04.939759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.939594    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 04 01:15:04.940040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:04.940002    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 04 01:15:05.207927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.207749    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:05.575070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.574694    5797 generic.go:356] "Generic (PLEG): container finished" podID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerID="42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2" exitCode=60
Nov 04 01:15:05.575070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.574809    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerDied","Data":"42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2"}
Nov 04 01:15:05.575070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.574865    5797 scope.go:122] "RemoveContainer" containerID="720d2b7644d4c5d0290f1ff2630a84afbe57689255233fb7cdbef537683d03d8"
Nov 04 01:15:05.588014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.585420    5797 scope.go:122] "RemoveContainer" containerID="42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2"
Nov 04 01:15:05.588014 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:05.585789    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r_odh-ai-gateway-infra(bf7b61f6-bb9f-4051-a743-bebeed856a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e"
Nov 04 01:15:05.604253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.603048    5797 generic.go:356] "Generic (PLEG): container finished" podID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerID="5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc" exitCode=60
Nov 04 01:15:05.604253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.603162    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerDied","Data":"5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc"}
Nov 04 01:15:05.604253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.603612    5797 scope.go:122] "RemoveContainer" containerID="5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc"
Nov 04 01:15:05.604253 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:05.603953    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx_odh-ai-gateway-infra(cd6aad7a-097e-4e2c-9a09-49c9020d94fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa"
Nov 04 01:15:05.607754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.607727    5797 generic.go:356] "Generic (PLEG): container finished" podID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerID="e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4" exitCode=60
Nov 04 01:15:05.607886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.607841    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" event={"ID":"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9","Type":"ContainerDied","Data":"e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4"}
Nov 04 01:15:05.608232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.608189    5797 scope.go:122] "RemoveContainer" containerID="e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4"
Nov 04 01:15:05.608554 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:05.608521    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml_odh-ai-gateway-infra(e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9"
Nov 04 01:15:05.610911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.610887    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerStarted","Data":"e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db"}
Nov 04 01:15:05.613562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.613545    5797 generic.go:356] "Generic (PLEG): container finished" podID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerID="9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43" exitCode=60
Nov 04 01:15:05.613838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.613818    5797 scope.go:122] "RemoveContainer" containerID="9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43"
Nov 04 01:15:05.615232 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:05.614106    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx_odh-ai-gateway-infra(7932a793-995c-4f0c-9d4e-51cc775b8b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54"
Nov 04 01:15:05.615232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.614945    5797 scope.go:122] "RemoveContainer" containerID="759a5760186c67073f9d77cf0225758cd693586c24117d4c60b762b8c7fefed9"
Nov 04 01:15:05.615232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.615052    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerDied","Data":"9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43"}
Nov 04 01:15:05.629733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.629674    5797 scope.go:122] "RemoveContainer" containerID="5f40545a069abbc32219f5ac40f4da24557e47259de888aa20ab4ddd7fe21fcb"
Nov 04 01:15:05.690037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.689050    5797 scope.go:122] "RemoveContainer" containerID="8e3582fcd382321ba33c1163beee747b77b347b6a0a1f1c78a09994e3b33a75d"
Nov 04 01:15:05.842070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.841968    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 04 01:15:05.842283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.842253    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 04 01:15:05.885750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.885706    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"]
Nov 04 01:15:05.919408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.919146    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"]
Nov 04 01:15:05.919408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.919281    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:05.923949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.923477    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:15:05.932798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.932770    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 04 01:15:05.932933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:05.932823    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 04 01:15:06.034005 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.033949    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcpb\" (UniqueName: \"kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.034179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.034018    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.034179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.034071    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.134971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.134933    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.135177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.135087    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcpb\" (UniqueName: \"kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.135177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.135120    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.135786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.135760    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.144392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.144365    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.170015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.166884    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcpb\" (UniqueName: \"kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb\") pod \"maas-api-revoke-sub-f012d617-lb7xh\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.236637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.236597    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:06.304017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.301832    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"]
Nov 04 01:15:06.316860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.316832    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.421940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.421906    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"]
Nov 04 01:15:06.439045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.438959    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlb2\" (UniqueName: \"kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.439190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.439046    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.439258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.439241    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.509165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.509088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:06.541045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.540971    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlb2\" (UniqueName: \"kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.541946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.541427    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.541946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.541654    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.542480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.542458    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.544283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.544262    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.596031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.571758    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlb2\" (UniqueName: \"kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2\") pod \"maas-api-revoke-sub-a778bb3e-6tpsq\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.610842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.610382    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"]
Nov 04 01:15:06.623221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.623199    5797 generic.go:356] "Generic (PLEG): container finished" podID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerID="e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db" exitCode=60
Nov 04 01:15:06.623393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.623377    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerDied","Data":"e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db"}
Nov 04 01:15:06.623487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.623478    5797 scope.go:122] "RemoveContainer" containerID="2716e8448b1f23089dc491886b061befd82565ce773725599c333c461d81cc59"
Nov 04 01:15:06.623880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.623866    5797 scope.go:122] "RemoveContainer" containerID="e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db"
Nov 04 01:15:06.624252 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:06.624232    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822_odh-ai-gateway-infra(0cc48035-c08c-41f2-91eb-7bd56d7deba3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3"
Nov 04 01:15:06.631794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.631771    5797 scope.go:122] "RemoveContainer" containerID="9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43"
Nov 04 01:15:06.632059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.632035    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh" event={"ID":"f42d7cdd-3302-40d0-b5b4-98187912f57a","Type":"ContainerStarted","Data":"9541b3572f624405aa9dae3df2caf2d5114afb0345a60a019fc386014e6d14b2"}
Nov 04 01:15:06.632181 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:06.632106    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx_odh-ai-gateway-infra(7932a793-995c-4f0c-9d4e-51cc775b8b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54"
Nov 04 01:15:06.632560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.632542    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:06.634924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.634898    5797 scope.go:122] "RemoveContainer" containerID="42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2"
Nov 04 01:15:06.635289 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:06.635186    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r_odh-ai-gateway-infra(bf7b61f6-bb9f-4051-a743-bebeed856a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e"
Nov 04 01:15:06.637333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.637315    5797 scope.go:122] "RemoveContainer" containerID="5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc"
Nov 04 01:15:06.639231 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:06.639207    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx_odh-ai-gateway-infra(cd6aad7a-097e-4e2c-9a09-49c9020d94fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa"
Nov 04 01:15:06.694219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.679387    5797 scope.go:122] "RemoveContainer" containerID="e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4"
Nov 04 01:15:06.694219 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:06.679636    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml_odh-ai-gateway-infra(e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9"
Nov 04 01:15:06.742034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.740458    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:06.842512 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.842481    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 04 01:15:06.842685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.842530    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 04 01:15:06.933765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.932889    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 04 01:15:06.933765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:06.932947    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 04 01:15:07.204116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.203980    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"]
Nov 04 01:15:07.206603 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:07.206565    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556cf0e8_60c6_4bb4_b02a_e26de0b56c32.slice/crio-bafa05abf00bff8531cb9e88de751c231e3cfd07b8812081275fdaecc33b5247 WatchSource:0}: Error finding container bafa05abf00bff8531cb9e88de751c231e3cfd07b8812081275fdaecc33b5247: Status 404 returned error can't find the container with id bafa05abf00bff8531cb9e88de751c231e3cfd07b8812081275fdaecc33b5247
Nov 04 01:15:07.576695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.576609    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:07.664038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.660651    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh" event={"ID":"f42d7cdd-3302-40d0-b5b4-98187912f57a","Type":"ContainerStarted","Data":"3db0849f0a39c61d3603c715701fee3d19a7b1dc45a9d0cccc0cdad5d806d5d4"}
Nov 04 01:15:07.686043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.683681    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh" podStartSLOduration=2.683662387 podStartE2EDuration="2.683662387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:07.682704101 +0000 UTC m=+695.324610355" watchObservedRunningTime="2026-11-04 01:15:07.683662387 +0000 UTC m=+695.325568638"
Nov 04 01:15:07.706013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.705308    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq" event={"ID":"556cf0e8-60c6-4bb4-b02a-e26de0b56c32","Type":"ContainerStarted","Data":"bafa05abf00bff8531cb9e88de751c231e3cfd07b8812081275fdaecc33b5247"}
Nov 04 01:15:07.708784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.708761    5797 scope.go:122] "RemoveContainer" containerID="e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db"
Nov 04 01:15:07.709134 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:07.709105    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822_odh-ai-gateway-infra(0cc48035-c08c-41f2-91eb-7bd56d7deba3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3"
Nov 04 01:15:07.842044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.841956    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 04 01:15:07.842260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.842233    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 04 01:15:07.932797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.932757    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 04 01:15:07.932941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:07.932827    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 04 01:15:08.541288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.541256    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:08.716835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.716802    5797 generic.go:356] "Generic (PLEG): container finished" podID="f42d7cdd-3302-40d0-b5b4-98187912f57a" containerID="3db0849f0a39c61d3603c715701fee3d19a7b1dc45a9d0cccc0cdad5d806d5d4" exitCode=0
Nov 04 01:15:08.717336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.717053    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh" event={"ID":"f42d7cdd-3302-40d0-b5b4-98187912f57a","Type":"ContainerDied","Data":"3db0849f0a39c61d3603c715701fee3d19a7b1dc45a9d0cccc0cdad5d806d5d4"}
Nov 04 01:15:08.718645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.718623    5797 generic.go:356] "Generic (PLEG): container finished" podID="556cf0e8-60c6-4bb4-b02a-e26de0b56c32" containerID="201ac6659cd295f323db3532b0981e3a7ddcb8c711919e029fea1ef57200579d" exitCode=0
Nov 04 01:15:08.718734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.718720    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq" event={"ID":"556cf0e8-60c6-4bb4-b02a-e26de0b56c32","Type":"ContainerDied","Data":"201ac6659cd295f323db3532b0981e3a7ddcb8c711919e029fea1ef57200579d"}
Nov 04 01:15:08.842094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.842058    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 04 01:15:08.842283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.842119    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 04 01:15:08.932475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.932433    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 04 01:15:08.932648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:08.932491    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 04 01:15:09.403957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:09.403915    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:09.841937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:09.841896    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 04 01:15:09.842449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:09.841969    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 04 01:15:09.936755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:09.933658    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 04 01:15:09.936755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:09.933730    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 04 01:15:09.984975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:09.984602    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:10.026838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.026803    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:10.136065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136035    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca\") pod \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") "
Nov 04 01:15:10.136234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136090    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb\" (UniqueName: \"kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb\") pod \"f42d7cdd-3302-40d0-b5b4-98187912f57a\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") "
Nov 04 01:15:10.136234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136126    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca\") pod \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") "
Nov 04 01:15:10.136234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136146    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca\") pod \"f42d7cdd-3302-40d0-b5b4-98187912f57a\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") "
Nov 04 01:15:10.136391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136242    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2\" (UniqueName: \"kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2\") pod \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\" (UID: \"556cf0e8-60c6-4bb4-b02a-e26de0b56c32\") "
Nov 04 01:15:10.136391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136266    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca\") pod \"f42d7cdd-3302-40d0-b5b4-98187912f57a\" (UID: \"f42d7cdd-3302-40d0-b5b4-98187912f57a\") "
Nov 04 01:15:10.136582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136550    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca" pod "556cf0e8-60c6-4bb4-b02a-e26de0b56c32" (UID: "556cf0e8-60c6-4bb4-b02a-e26de0b56c32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:10.136678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.136592    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca" pod "f42d7cdd-3302-40d0-b5b4-98187912f57a" (UID: "f42d7cdd-3302-40d0-b5b4-98187912f57a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:10.137260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.137236    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:10.137365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.137265    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:10.138449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.138422    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb" pod "f42d7cdd-3302-40d0-b5b4-98187912f57a" (UID: "f42d7cdd-3302-40d0-b5b4-98187912f57a"). InnerVolumeSpecName "kube-api-access-qwcpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:10.138611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.138588    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca" pod "f42d7cdd-3302-40d0-b5b4-98187912f57a" (UID: "f42d7cdd-3302-40d0-b5b4-98187912f57a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:10.139221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.139200    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2" pod "556cf0e8-60c6-4bb4-b02a-e26de0b56c32" (UID: "556cf0e8-60c6-4bb4-b02a-e26de0b56c32"). InnerVolumeSpecName "kube-api-access-bxlb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:10.139539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.139514    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca" pod "556cf0e8-60c6-4bb4-b02a-e26de0b56c32" (UID: "556cf0e8-60c6-4bb4-b02a-e26de0b56c32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:10.238539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.238453    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcpb\" (UniqueName: \"kubernetes.io/projected/f42d7cdd-3302-40d0-b5b4-98187912f57a-kube-api-access-qwcpb\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:10.238539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.238491    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlb2\" (UniqueName: \"kubernetes.io/projected/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-kube-api-access-bxlb2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:10.238539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.238511    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42d7cdd-3302-40d0-b5b4-98187912f57a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:10.238539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.238527    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556cf0e8-60c6-4bb4-b02a-e26de0b56c32-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:10.439852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.439800    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:10.627398 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.627350    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"]
Nov 04 01:15:10.628574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.628542    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f42d7cdd-3302-40d0-b5b4-98187912f57a" containerName="revoke-keys"
Nov 04 01:15:10.628574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.628573    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42d7cdd-3302-40d0-b5b4-98187912f57a" containerName="revoke-keys"
Nov 04 01:15:10.628772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.628646    5797 cpu_manager.go:414] "containerMap: removing container" podUID="556cf0e8-60c6-4bb4-b02a-e26de0b56c32" containerName="revoke-keys"
Nov 04 01:15:10.628772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.628653    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556cf0e8-60c6-4bb4-b02a-e26de0b56c32" containerName="revoke-keys"
Nov 04 01:15:10.628876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.628827    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="556cf0e8-60c6-4bb4-b02a-e26de0b56c32" containerName="revoke-keys"
Nov 04 01:15:10.628876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.628853    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42d7cdd-3302-40d0-b5b4-98187912f57a" containerName="revoke-keys"
Nov 04 01:15:10.671806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.671768    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.675389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.675330    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"]
Nov 04 01:15:10.699486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.698526    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1036e8\"/\"distinct-w1-1036e8-kserve-self-signed-certs\""
Nov 04 01:15:10.741021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.739250    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"
Nov 04 01:15:10.741021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.739256    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh" event={"ID":"f42d7cdd-3302-40d0-b5b4-98187912f57a","Type":"ContainerDied","Data":"9541b3572f624405aa9dae3df2caf2d5114afb0345a60a019fc386014e6d14b2"}
Nov 04 01:15:10.741021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.739302    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9541b3572f624405aa9dae3df2caf2d5114afb0345a60a019fc386014e6d14b2"
Nov 04 01:15:10.741021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.740909    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq" event={"ID":"556cf0e8-60c6-4bb4-b02a-e26de0b56c32","Type":"ContainerDied","Data":"bafa05abf00bff8531cb9e88de751c231e3cfd07b8812081275fdaecc33b5247"}
Nov 04 01:15:10.741021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.740942    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bafa05abf00bff8531cb9e88de751c231e3cfd07b8812081275fdaecc33b5247"
Nov 04 01:15:10.741021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.741003    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"
Nov 04 01:15:10.763772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.763681    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:10.854024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.853779    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.855241    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5mj\" (UniqueName: \"kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.855310    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.855351    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.855395    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.855461    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.855633    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:10.857016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.856286    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:10.857423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.857336    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.957376    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5mj\" (UniqueName: \"kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.957453    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.957492    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.957536    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.957599    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.957654    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.958229    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.958766    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.966217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.959427    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.967114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.967091    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.967435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.967419    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:10.969954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:10.969933    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5mj\" (UniqueName: \"kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj\") pod \"distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") " pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:11.014984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.014885    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:11.187354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.187323    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:11.188336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.188305    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:11.189361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.189339    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:15:11.318599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.318574    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"]
Nov 04 01:15:11.320147 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:11.320116    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29348c5f_115b_4823_9a13_fe0c34d6765d.slice/crio-beb2a8272545ae4e66b9b9a34cd9a9eea25a017e0cfa2fbafc93f797280c9455 WatchSource:0}: Error finding container beb2a8272545ae4e66b9b9a34cd9a9eea25a017e0cfa2fbafc93f797280c9455: Status 404 returned error can't find the container with id beb2a8272545ae4e66b9b9a34cd9a9eea25a017e0cfa2fbafc93f797280c9455
Nov 04 01:15:11.379161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.379134    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:11.751272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.751235    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" event={"ID":"29348c5f-115b-4823-9a13-fe0c34d6765d","Type":"ContainerStarted","Data":"df914983beeed3953e76d4ed81d2971ae6124bb8ea350379582b40d71ee6a3d2"}
Nov 04 01:15:11.751433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.751285    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" event={"ID":"29348c5f-115b-4823-9a13-fe0c34d6765d","Type":"ContainerStarted","Data":"beb2a8272545ae4e66b9b9a34cd9a9eea25a017e0cfa2fbafc93f797280c9455"}
Nov 04 01:15:11.772545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:11.772489    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podStartSLOduration=1.772473426 podStartE2EDuration="1.772473426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:11.770933705 +0000 UTC m=+699.412839983" watchObservedRunningTime="2026-11-04 01:15:11.772473426 +0000 UTC m=+699.414379678"
Nov 04 01:15:12.015734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:12.015665    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:12.207511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:12.207464    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:12.892134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:12.892105    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:13.768529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:13.768491    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:14.518048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:14.502516    5797 scope.go:122] "RemoveContainer" containerID="e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4"
Nov 04 01:15:14.736009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:14.734951    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:15.471191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.471162    5797 scope.go:122] "RemoveContainer" containerID="42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2"
Nov 04 01:15:15.794689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.794658    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:15.828081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.828049    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerStarted","Data":"3c0a929ec632070907e1384534eb292356f8eb32802860769d20db904d4850d2"}
Nov 04 01:15:15.830015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.829967    5797 generic.go:356] "Generic (PLEG): container finished" podID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerID="3fb190ccf3f62fc5ab2c9446a67ce0f0055eb070d73241ba4695ca6365e0089a" exitCode=60
Nov 04 01:15:15.830145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.830040    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" event={"ID":"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9","Type":"ContainerDied","Data":"3fb190ccf3f62fc5ab2c9446a67ce0f0055eb070d73241ba4695ca6365e0089a"}
Nov 04 01:15:15.830145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.830091    5797 scope.go:122] "RemoveContainer" containerID="e2efd9f9aef9cb3c46c26d0ecbe0a07a6ccd89ce57bd1c83651b24891b281fe4"
Nov 04 01:15:15.830438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:15.830422    5797 scope.go:122] "RemoveContainer" containerID="3fb190ccf3f62fc5ab2c9446a67ce0f0055eb070d73241ba4695ca6365e0089a"
Nov 04 01:15:15.830769 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:15.830732    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml_odh-ai-gateway-infra(e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9"
Nov 04 01:15:16.471405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.471377    5797 scope.go:122] "RemoveContainer" containerID="5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc"
Nov 04 01:15:16.471864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.471470    5797 scope.go:122] "RemoveContainer" containerID="9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43"
Nov 04 01:15:16.656093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.656061    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:16.838951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.838921    5797 generic.go:356] "Generic (PLEG): container finished" podID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerID="3c0a929ec632070907e1384534eb292356f8eb32802860769d20db904d4850d2" exitCode=60
Nov 04 01:15:16.838951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.838941    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerDied","Data":"3c0a929ec632070907e1384534eb292356f8eb32802860769d20db904d4850d2"}
Nov 04 01:15:16.839208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.839015    5797 scope.go:122] "RemoveContainer" containerID="42d724c4b4d7e35c02473c177645e2cf7f91670010cb999b0b0b6393fbaf6ea2"
Nov 04 01:15:16.839387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.839365    5797 scope.go:122] "RemoveContainer" containerID="3c0a929ec632070907e1384534eb292356f8eb32802860769d20db904d4850d2"
Nov 04 01:15:16.839699 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:16.839676    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r_odh-ai-gateway-infra(bf7b61f6-bb9f-4051-a743-bebeed856a7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e"
Nov 04 01:15:16.841149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.841125    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerStarted","Data":"1c47d4f9eddded56eaf5d0188515112efb02f97e85799ed988bc0fcf882306e7"}
Nov 04 01:15:16.844870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.844850    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerStarted","Data":"8902d075f34c471404448ffcd29e111684d9715b7e4adf207d95420e4a1046aa"}
Nov 04 01:15:16.859393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.859370    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"]
Nov 04 01:15:16.880690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:16.880665    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"]
Nov 04 01:15:17.025552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.025493    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:17.080558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.079958    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7\" (UniqueName: \"kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7\") pod \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") "
Nov 04 01:15:17.080558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.080199    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca\") pod \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\" (UID: \"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9\") "
Nov 04 01:15:17.081468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.081147    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca" pod "e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" (UID: "e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:17.083251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.083224    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7" pod "e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" (UID: "e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9"). InnerVolumeSpecName "kube-api-access-smwp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:17.182689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.182652    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:17.182689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.182686    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwp7\" (UniqueName: \"kubernetes.io/projected/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9-kube-api-access-smwp7\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:17.472244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.471662    5797 scope.go:122] "RemoveContainer" containerID="e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db"
Nov 04 01:15:17.493698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.493650    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:17.855325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.855292    5797 generic.go:356] "Generic (PLEG): container finished" podID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerID="1c47d4f9eddded56eaf5d0188515112efb02f97e85799ed988bc0fcf882306e7" exitCode=60
Nov 04 01:15:17.855439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.855359    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerDied","Data":"1c47d4f9eddded56eaf5d0188515112efb02f97e85799ed988bc0fcf882306e7"}
Nov 04 01:15:17.855439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.855419    5797 scope.go:122] "RemoveContainer" containerID="5e047e5b8ff87632b6c4847d4b9b297b009cfee3588dddbaf95ce0e39bf159cc"
Nov 04 01:15:17.855802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.855785    5797 scope.go:122] "RemoveContainer" containerID="1c47d4f9eddded56eaf5d0188515112efb02f97e85799ed988bc0fcf882306e7"
Nov 04 01:15:17.856162 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:17.856127    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx_odh-ai-gateway-infra(cd6aad7a-097e-4e2c-9a09-49c9020d94fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa"
Nov 04 01:15:17.857128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.857111    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"
Nov 04 01:15:17.857263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.857154    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml" event={"ID":"e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9","Type":"ContainerDied","Data":"25d3606910f08f90a3f09e3b82d0e7a581608bfb6972f017696a8f07ce3775b4"}
Nov 04 01:15:17.859479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.859455    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerStarted","Data":"f99e54616ff0a47817538fb466a9f51f95db4a867633bb1ce5f24c67d2351f9c"}
Nov 04 01:15:17.861898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.861875    5797 generic.go:356] "Generic (PLEG): container finished" podID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerID="8902d075f34c471404448ffcd29e111684d9715b7e4adf207d95420e4a1046aa" exitCode=60
Nov 04 01:15:17.862047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.861944    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerDied","Data":"8902d075f34c471404448ffcd29e111684d9715b7e4adf207d95420e4a1046aa"}
Nov 04 01:15:17.862411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.862392    5797 scope.go:122] "RemoveContainer" containerID="8902d075f34c471404448ffcd29e111684d9715b7e4adf207d95420e4a1046aa"
Nov 04 01:15:17.862677 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:17.862651    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx_odh-ai-gateway-infra(7932a793-995c-4f0c-9d4e-51cc775b8b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54"
Nov 04 01:15:17.871008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.870916    5797 scope.go:122] "RemoveContainer" containerID="3fb190ccf3f62fc5ab2c9446a67ce0f0055eb070d73241ba4695ca6365e0089a"
Nov 04 01:15:17.881443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.881418    5797 scope.go:122] "RemoveContainer" containerID="9cfb3b4ffe965137534a933340195c3f06be689a7384cbabd9fa798cbedc9e43"
Nov 04 01:15:17.909075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.909048    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"]
Nov 04 01:15:17.919197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.917416    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"]
Nov 04 01:15:17.924327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.924299    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6dac99-29895915-svwml"]
Nov 04 01:15:17.929651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:17.929622    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"]
Nov 04 01:15:18.018270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.018245    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:18.093486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.093453    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh\" (UniqueName: \"kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh\") pod \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") "
Nov 04 01:15:18.093646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.093551    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca\") pod \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\" (UID: \"bf7b61f6-bb9f-4051-a743-bebeed856a7e\") "
Nov 04 01:15:18.093908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.093886    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca" pod "bf7b61f6-bb9f-4051-a743-bebeed856a7e" (UID: "bf7b61f6-bb9f-4051-a743-bebeed856a7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:18.095518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.095494    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh" pod "bf7b61f6-bb9f-4051-a743-bebeed856a7e" (UID: "bf7b61f6-bb9f-4051-a743-bebeed856a7e"). InnerVolumeSpecName "kube-api-access-cdshh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:18.195025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.194932    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdshh\" (UniqueName: \"kubernetes.io/projected/bf7b61f6-bb9f-4051-a743-bebeed856a7e-kube-api-access-cdshh\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:18.195025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.194961    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b61f6-bb9f-4051-a743-bebeed856a7e-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:18.447723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.447622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:18.480129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.480096    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" path="/var/lib/kubelet/pods/e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9/volumes"
Nov 04 01:15:18.709301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.709222    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"]
Nov 04 01:15:18.710155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710130    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710157    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710176    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:18.710278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710191    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:18.710278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710221    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:18.710278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710229    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:18.710631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710300    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710309    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710798    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:18.710864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710836    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710849    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710859    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:18.710981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.710868    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:18.715319 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.715296    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.720598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.720574    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:15:18.738496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.738464    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"]
Nov 04 01:15:18.806826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.806780    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.807026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.806906    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjl49\" (UniqueName: \"kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.807026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.806981    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.888023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.886161    5797 generic.go:356] "Generic (PLEG): container finished" podID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerID="f99e54616ff0a47817538fb466a9f51f95db4a867633bb1ce5f24c67d2351f9c" exitCode=60
Nov 04 01:15:18.888023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.886252    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerDied","Data":"f99e54616ff0a47817538fb466a9f51f95db4a867633bb1ce5f24c67d2351f9c"}
Nov 04 01:15:18.888023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.886306    5797 scope.go:122] "RemoveContainer" containerID="e8255e8c7a11e4c45968f991fd779ab7704c6277c4b2d20d8efbd656c2e228db"
Nov 04 01:15:18.888023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.886775    5797 scope.go:122] "RemoveContainer" containerID="f99e54616ff0a47817538fb466a9f51f95db4a867633bb1ce5f24c67d2351f9c"
Nov 04 01:15:18.888023 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:18.887136    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822_odh-ai-gateway-infra(0cc48035-c08c-41f2-91eb-7bd56d7deba3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3"
Nov 04 01:15:18.894582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.894549    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r" event={"ID":"bf7b61f6-bb9f-4051-a743-bebeed856a7e","Type":"ContainerDied","Data":"d689dcd3741d940cd9753f1d2aa73faf10f75b7fe4a62008c40cfa2b66500e11"}
Nov 04 01:15:18.894697 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.894672    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"
Nov 04 01:15:18.908090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.908053    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.908250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.908233    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.908368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.908328    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjl49\" (UniqueName: \"kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.910197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.909150    5797 scope.go:122] "RemoveContainer" containerID="3c0a929ec632070907e1384534eb292356f8eb32802860769d20db904d4850d2"
Nov 04 01:15:18.910197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.910103    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.921942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.918456    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.925565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.925526    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjl49\" (UniqueName: \"kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49\") pod \"maas-api-revoke-sub-8131eae8-jcdfr\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:18.929245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.929214    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"]
Nov 04 01:15:18.935008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:18.932856    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895915-d586r"]
Nov 04 01:15:19.034445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.034406    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:19.084072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.082558    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"]
Nov 04 01:15:19.095801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.095773    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:19.155512 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.155487    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"]
Nov 04 01:15:19.156607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.156583    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:19.161695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.161672    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" containerName="cleanup"
Nov 04 01:15:19.161842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.161829    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.161931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.161919    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.168088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168054    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:19.168088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168077    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:19.168088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168093    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.168346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168102    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.168346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168330    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d0ab1e-b4ac-4ed8-88ae-d8532e7220f9" containerName="cleanup"
Nov 04 01:15:19.168346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168347    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.168676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.168362    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.189949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.174459    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.189949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.179913    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-875528\"/\"distinct-w5-875528-kserve-self-signed-certs\""
Nov 04 01:15:19.189949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.187717    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"]
Nov 04 01:15:19.191924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.191899    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"]
Nov 04 01:15:19.192775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.192753    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.192918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.192902    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.194013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.193979    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" containerName="cleanup"
Nov 04 01:15:19.196392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.196324    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:19.200211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.200187    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"]
Nov 04 01:15:19.200322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.200295    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.212890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.212869    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc\" (UniqueName: \"kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc\") pod \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") "
Nov 04 01:15:19.213476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.213457    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca\") pod \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\" (UID: \"cd6aad7a-097e-4e2c-9a09-49c9020d94fa\") "
Nov 04 01:15:19.214625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.214504    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca" pod "cd6aad7a-097e-4e2c-9a09-49c9020d94fa" (UID: "cd6aad7a-097e-4e2c-9a09-49c9020d94fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:19.215128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.215104    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.215365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.215349    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.216411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.215833    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svv6v\" (UniqueName: \"kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.216411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.215879    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.216411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.216204    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.216411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.216253    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.216411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.216334    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:19.221163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.221140    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc" pod "cd6aad7a-097e-4e2c-9a09-49c9020d94fa" (UID: "cd6aad7a-097e-4e2c-9a09-49c9020d94fa"). InnerVolumeSpecName "kube-api-access-xpfkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:19.295538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.295495    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:19.332039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.320234    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz\" (UniqueName: \"kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz\") pod \"7932a793-995c-4f0c-9d4e-51cc775b8b54\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") "
Nov 04 01:15:19.332413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332109    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca\") pod \"7932a793-995c-4f0c-9d4e-51cc775b8b54\" (UID: \"7932a793-995c-4f0c-9d4e-51cc775b8b54\") "
Nov 04 01:15:19.332582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332505    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.332582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332548    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332591    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzlv\" (UniqueName: \"kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332629    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332665    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332702    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svv6v\" (UniqueName: \"kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332725    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332752    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.332797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332778    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.333260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.332897    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfkc\" (UniqueName: \"kubernetes.io/projected/cd6aad7a-097e-4e2c-9a09-49c9020d94fa-kube-api-access-xpfkc\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:19.334455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.334047    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca" pod "7932a793-995c-4f0c-9d4e-51cc775b8b54" (UID: "7932a793-995c-4f0c-9d4e-51cc775b8b54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:19.334455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.334345    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.334802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.334780    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.341333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.335527    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.341333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.336366    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz" pod "7932a793-995c-4f0c-9d4e-51cc775b8b54" (UID: "7932a793-995c-4f0c-9d4e-51cc775b8b54"). InnerVolumeSpecName "kube-api-access-xl7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:19.341333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.338067    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.341333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.338338    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.348335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.348261    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svv6v\" (UniqueName: \"kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v\") pod \"distinct-w5-875528-kserve-77c8c8df7-brdl2\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") " pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.372065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.372032    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"]
Nov 04 01:15:19.434559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.434526    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzlv\" (UniqueName: \"kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.434728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.434642    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.434728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.434671    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.434845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.434758    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7xz\" (UniqueName: \"kubernetes.io/projected/7932a793-995c-4f0c-9d4e-51cc775b8b54-kube-api-access-xl7xz\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:19.434845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.434776    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7932a793-995c-4f0c-9d4e-51cc775b8b54-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:19.435510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.435482    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.437149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.437127    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.446654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.446633    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzlv\" (UniqueName: \"kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv\") pod \"maas-api-revoke-sub-849450a3-qvv6p\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.505336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.505264    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:19.517466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.517430    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:19.680214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.680180    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:19.793701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.793671    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"]
Nov 04 01:15:19.794406 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:19.794382    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8c17db7_cbad_41ff_92f4_58bc2d4a6458.slice/crio-69c5f9ac3d68b2127758d2cdb8334776ba5e49b4620afc51aedb91e3330459f5 WatchSource:0}: Error finding container 69c5f9ac3d68b2127758d2cdb8334776ba5e49b4620afc51aedb91e3330459f5: Status 404 returned error can't find the container with id 69c5f9ac3d68b2127758d2cdb8334776ba5e49b4620afc51aedb91e3330459f5
Nov 04 01:15:19.812946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.812919    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"]
Nov 04 01:15:19.813769 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:19.813746    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc7f129_bf83_42a9_b6f3_bb497180eed0.slice/crio-61285fe2a6a26e6c6848982451ba489aacfa2b6f7e93bd9d6c9115b45805858d WatchSource:0}: Error finding container 61285fe2a6a26e6c6848982451ba489aacfa2b6f7e93bd9d6c9115b45805858d: Status 404 returned error can't find the container with id 61285fe2a6a26e6c6848982451ba489aacfa2b6f7e93bd9d6c9115b45805858d
Nov 04 01:15:19.904244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.904216    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"
Nov 04 01:15:19.904419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.904218    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx" event={"ID":"cd6aad7a-097e-4e2c-9a09-49c9020d94fa","Type":"ContainerDied","Data":"b0cc90ef801c7f0b8e33bfbd837c9520e62ce6470e53fecee84ec7b900a73ce1"}
Nov 04 01:15:19.904419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.904361    5797 scope.go:122] "RemoveContainer" containerID="1c47d4f9eddded56eaf5d0188515112efb02f97e85799ed988bc0fcf882306e7"
Nov 04 01:15:19.906629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.906604    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" event={"ID":"d8c17db7-cbad-41ff-92f4-58bc2d4a6458","Type":"ContainerStarted","Data":"82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf"}
Nov 04 01:15:19.906629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.906639    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" event={"ID":"d8c17db7-cbad-41ff-92f4-58bc2d4a6458","Type":"ContainerStarted","Data":"69c5f9ac3d68b2127758d2cdb8334776ba5e49b4620afc51aedb91e3330459f5"}
Nov 04 01:15:19.908780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.908733    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr" event={"ID":"deb3ce1e-3f6b-4020-a280-845c70a362da","Type":"ContainerStarted","Data":"704171a5e92ca97b08a3b41377fd9d6a3825e3dd86cc3b6b25e8d74d8bd4a1c0"}
Nov 04 01:15:19.908875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.908807    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr" event={"ID":"deb3ce1e-3f6b-4020-a280-845c70a362da","Type":"ContainerStarted","Data":"7a35610df3c7e1bd0fcd265e71286cfacdf80fce4398772b511c19305eefeb63"}
Nov 04 01:15:19.912491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.912461    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx" event={"ID":"7932a793-995c-4f0c-9d4e-51cc775b8b54","Type":"ContainerDied","Data":"f08344f7ec0a1a24c7f079ea5bd424c24242681396a9f8714fa56c4034328a1e"}
Nov 04 01:15:19.912618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.912567    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"
Nov 04 01:15:19.918474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.918457    5797 scope.go:122] "RemoveContainer" containerID="8902d075f34c471404448ffcd29e111684d9715b7e4adf207d95420e4a1046aa"
Nov 04 01:15:19.918627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.918516    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p" event={"ID":"6dc7f129-bf83-42a9-b6f3-bb497180eed0","Type":"ContainerStarted","Data":"61285fe2a6a26e6c6848982451ba489aacfa2b6f7e93bd9d6c9115b45805858d"}
Nov 04 01:15:19.932881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.932831    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podStartSLOduration=0.932815007 podStartE2EDuration="932.815007ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:19.927753081 +0000 UTC m=+707.569659355" watchObservedRunningTime="2026-11-04 01:15:19.932815007 +0000 UTC m=+707.574721258"
Nov 04 01:15:19.946542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.946508    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"]
Nov 04 01:15:19.951931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.951904    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-875528-29895915-65pjx"]
Nov 04 01:15:19.990285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:19.990200    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr" podStartSLOduration=1.990174689 podStartE2EDuration="1.990174689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:19.987174269 +0000 UTC m=+707.629080571" watchObservedRunningTime="2026-11-04 01:15:19.990174689 +0000 UTC m=+707.632080941"
Nov 04 01:15:20.004553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.004518    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"]
Nov 04 01:15:20.008585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.008553    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9902c3-29895915-qcdnx"]
Nov 04 01:15:20.086256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.086228    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:20.146843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.146807    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca\") pod \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") "
Nov 04 01:15:20.147084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.146883    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh\" (UniqueName: \"kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh\") pod \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\" (UID: \"0cc48035-c08c-41f2-91eb-7bd56d7deba3\") "
Nov 04 01:15:20.148374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.148347    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca" pod "0cc48035-c08c-41f2-91eb-7bd56d7deba3" (UID: "0cc48035-c08c-41f2-91eb-7bd56d7deba3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:20.150296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.150250    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh" pod "0cc48035-c08c-41f2-91eb-7bd56d7deba3" (UID: "0cc48035-c08c-41f2-91eb-7bd56d7deba3"). InnerVolumeSpecName "kube-api-access-9qjhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:20.249162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.249130    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc48035-c08c-41f2-91eb-7bd56d7deba3-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:20.249162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.249158    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjhh\" (UniqueName: \"kubernetes.io/projected/0cc48035-c08c-41f2-91eb-7bd56d7deba3-kube-api-access-9qjhh\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:20.466144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.466114    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:20.478505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.478480    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" path="/var/lib/kubelet/pods/7932a793-995c-4f0c-9d4e-51cc775b8b54/volumes"
Nov 04 01:15:20.478900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.478886    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7b61f6-bb9f-4051-a743-bebeed856a7e" path="/var/lib/kubelet/pods/bf7b61f6-bb9f-4051-a743-bebeed856a7e/volumes"
Nov 04 01:15:20.479305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.479291    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6aad7a-097e-4e2c-9a09-49c9020d94fa" path="/var/lib/kubelet/pods/cd6aad7a-097e-4e2c-9a09-49c9020d94fa/volumes"
Nov 04 01:15:20.505961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.505939    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:20.930459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.930424    5797 generic.go:356] "Generic (PLEG): container finished" podID="deb3ce1e-3f6b-4020-a280-845c70a362da" containerID="704171a5e92ca97b08a3b41377fd9d6a3825e3dd86cc3b6b25e8d74d8bd4a1c0" exitCode=0
Nov 04 01:15:20.930658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.930508    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr" event={"ID":"deb3ce1e-3f6b-4020-a280-845c70a362da","Type":"ContainerDied","Data":"704171a5e92ca97b08a3b41377fd9d6a3825e3dd86cc3b6b25e8d74d8bd4a1c0"}
Nov 04 01:15:20.931758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.931726    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822" event={"ID":"0cc48035-c08c-41f2-91eb-7bd56d7deba3","Type":"ContainerDied","Data":"f1ce1259ac6d5009e3bcaeea343988b6a85591df950a00ef5f869e4a0a02af1c"}
Nov 04 01:15:20.931868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.931782    5797 scope.go:122] "RemoveContainer" containerID="f99e54616ff0a47817538fb466a9f51f95db4a867633bb1ce5f24c67d2351f9c"
Nov 04 01:15:20.931868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.931739    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"
Nov 04 01:15:20.935212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.934493    5797 generic.go:356] "Generic (PLEG): container finished" podID="6dc7f129-bf83-42a9-b6f3-bb497180eed0" containerID="810dcdc2f8cdd99241a3d36fc36a5c2dda356fad7eae45a6d4b3a3942baf3efc" exitCode=0
Nov 04 01:15:20.935212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.934671    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p" event={"ID":"6dc7f129-bf83-42a9-b6f3-bb497180eed0","Type":"ContainerDied","Data":"810dcdc2f8cdd99241a3d36fc36a5c2dda356fad7eae45a6d4b3a3942baf3efc"}
Nov 04 01:15:20.983274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.983245    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"]
Nov 04 01:15:20.987695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:20.987664    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cffa80-29895915-hr822"]
Nov 04 01:15:21.015393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:21.015371    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:21.029117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:21.029087    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:21.182793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:21.182713    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:21.957494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:21.957460    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:15:21.991981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:21.991688    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:22.171199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.171177    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:22.173258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.173239    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:22.274985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.274908    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca\") pod \"deb3ce1e-3f6b-4020-a280-845c70a362da\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") "
Nov 04 01:15:22.275134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275041    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv\" (UniqueName: \"kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv\") pod \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") "
Nov 04 01:15:22.275202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275157    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca\") pod \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") "
Nov 04 01:15:22.275202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275188    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca\") pod \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\" (UID: \"6dc7f129-bf83-42a9-b6f3-bb497180eed0\") "
Nov 04 01:15:22.275286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275264    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca\") pod \"deb3ce1e-3f6b-4020-a280-845c70a362da\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") "
Nov 04 01:15:22.275335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275292    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49\" (UniqueName: \"kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49\") pod \"deb3ce1e-3f6b-4020-a280-845c70a362da\" (UID: \"deb3ce1e-3f6b-4020-a280-845c70a362da\") "
Nov 04 01:15:22.275658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275634    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca" pod "6dc7f129-bf83-42a9-b6f3-bb497180eed0" (UID: "6dc7f129-bf83-42a9-b6f3-bb497180eed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:22.275910 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275880    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:22.276066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.275978    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca" pod "deb3ce1e-3f6b-4020-a280-845c70a362da" (UID: "deb3ce1e-3f6b-4020-a280-845c70a362da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:22.277284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.277262    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv" pod "6dc7f129-bf83-42a9-b6f3-bb497180eed0" (UID: "6dc7f129-bf83-42a9-b6f3-bb497180eed0"). InnerVolumeSpecName "kube-api-access-8lzlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:22.277403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.277331    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca" pod "6dc7f129-bf83-42a9-b6f3-bb497180eed0" (UID: "6dc7f129-bf83-42a9-b6f3-bb497180eed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:22.277403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.277381    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca" pod "deb3ce1e-3f6b-4020-a280-845c70a362da" (UID: "deb3ce1e-3f6b-4020-a280-845c70a362da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:22.277613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.277598    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49" pod "deb3ce1e-3f6b-4020-a280-845c70a362da" (UID: "deb3ce1e-3f6b-4020-a280-845c70a362da"). InnerVolumeSpecName "kube-api-access-pjl49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:22.377636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.377603    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzlv\" (UniqueName: \"kubernetes.io/projected/6dc7f129-bf83-42a9-b6f3-bb497180eed0-kube-api-access-8lzlv\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:22.377636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.377633    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc7f129-bf83-42a9-b6f3-bb497180eed0-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:22.377636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.377643    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:22.377891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.377652    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjl49\" (UniqueName: \"kubernetes.io/projected/deb3ce1e-3f6b-4020-a280-845c70a362da-kube-api-access-pjl49\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:22.377891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.377661    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb3ce1e-3f6b-4020-a280-845c70a362da-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:22.478797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.478767    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" path="/var/lib/kubelet/pods/0cc48035-c08c-41f2-91eb-7bd56d7deba3/volumes"
Nov 04 01:15:22.951128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.951088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:22.962008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.954725    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr" event={"ID":"deb3ce1e-3f6b-4020-a280-845c70a362da","Type":"ContainerDied","Data":"7a35610df3c7e1bd0fcd265e71286cfacdf80fce4398772b511c19305eefeb63"}
Nov 04 01:15:22.962008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.954800    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a35610df3c7e1bd0fcd265e71286cfacdf80fce4398772b511c19305eefeb63"
Nov 04 01:15:22.962008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.954925    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"
Nov 04 01:15:22.964083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.964059    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"
Nov 04 01:15:22.964083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.964061    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p" event={"ID":"6dc7f129-bf83-42a9-b6f3-bb497180eed0","Type":"ContainerDied","Data":"61285fe2a6a26e6c6848982451ba489aacfa2b6f7e93bd9d6c9115b45805858d"}
Nov 04 01:15:22.964300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:22.964106    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61285fe2a6a26e6c6848982451ba489aacfa2b6f7e93bd9d6c9115b45805858d"
Nov 04 01:15:23.318269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318192    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"]
Nov 04 01:15:23.318753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318739    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.318795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318754    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.318795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318772    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.318795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318779    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.318795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318787    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:23.318795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318792    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318803    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318808    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318820    5797 cpu_manager.go:414] "containerMap: removing container" podUID="deb3ce1e-3f6b-4020-a280-845c70a362da" containerName="revoke-keys"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318825    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb3ce1e-3f6b-4020-a280-845c70a362da" containerName="revoke-keys"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318831    5797 cpu_manager.go:414] "containerMap: removing container" podUID="6dc7f129-bf83-42a9-b6f3-bb497180eed0" containerName="revoke-keys"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318836    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc7f129-bf83-42a9-b6f3-bb497180eed0" containerName="revoke-keys"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318916    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318923    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318933    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.318942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318940    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:23.319230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318947    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:23.319230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318954    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb3ce1e-3f6b-4020-a280-845c70a362da" containerName="revoke-keys"
Nov 04 01:15:23.319230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.318959    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc7f129-bf83-42a9-b6f3-bb497180eed0" containerName="revoke-keys"
Nov 04 01:15:23.323697 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.323674    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.327921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.327895    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:15:23.328060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.327920    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:15:23.331459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.331428    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"]
Nov 04 01:15:23.391109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.391074    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.391109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.391120    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.391547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.391375    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pb6\" (UniqueName: \"kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.492224 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.492190    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pb6\" (UniqueName: \"kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.492392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.492279    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.492392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.492315    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.493043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.493008    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.494712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.494689    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.505566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.505545    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pb6\" (UniqueName: \"kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6\") pod \"maas-api-revoke-sub-5c620916-p8c4s\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.558898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.556460    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:23.648247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.648215    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:23.872884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.872851    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:23.932909 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.932880    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"]
Nov 04 01:15:23.933589 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:23.933561    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e1decf_a43d_4327_9936_c9f34af0e317.slice/crio-3a595f16b10e5b714052b1432a4e39c920d8ef660a08f7b68c8d160209872823 WatchSource:0}: Error finding container 3a595f16b10e5b714052b1432a4e39c920d8ef660a08f7b68c8d160209872823: Status 404 returned error can't find the container with id 3a595f16b10e5b714052b1432a4e39c920d8ef660a08f7b68c8d160209872823
Nov 04 01:15:23.971134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:23.971110    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s" event={"ID":"14e1decf-a43d-4327-9936-c9f34af0e317","Type":"ContainerStarted","Data":"3a595f16b10e5b714052b1432a4e39c920d8ef660a08f7b68c8d160209872823"}
Nov 04 01:15:24.671123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:24.671091    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:24.981187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:24.981101    5797 generic.go:356] "Generic (PLEG): container finished" podID="14e1decf-a43d-4327-9936-c9f34af0e317" containerID="94120c38e9174fd3a1619963b4cb0bd9ccbc808ef87dea1127557c55cc779319" exitCode=0
Nov 04 01:15:24.981535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:24.981185    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s" event={"ID":"14e1decf-a43d-4327-9936-c9f34af0e317","Type":"ContainerDied","Data":"94120c38e9174fd3a1619963b4cb0bd9ccbc808ef87dea1127557c55cc779319"}
Nov 04 01:15:25.484215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:25.484185    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:26.158019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.157977    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:26.173934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.173907    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:26.226107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.226077    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6\" (UniqueName: \"kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6\") pod \"14e1decf-a43d-4327-9936-c9f34af0e317\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") "
Nov 04 01:15:26.226292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.226167    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca\") pod \"14e1decf-a43d-4327-9936-c9f34af0e317\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") "
Nov 04 01:15:26.226370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.226353    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca\") pod \"14e1decf-a43d-4327-9936-c9f34af0e317\" (UID: \"14e1decf-a43d-4327-9936-c9f34af0e317\") "
Nov 04 01:15:26.226849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.226807    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca" pod "14e1decf-a43d-4327-9936-c9f34af0e317" (UID: "14e1decf-a43d-4327-9936-c9f34af0e317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:26.228802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.228774    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6" pod "14e1decf-a43d-4327-9936-c9f34af0e317" (UID: "14e1decf-a43d-4327-9936-c9f34af0e317"). InnerVolumeSpecName "kube-api-access-k9pb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:26.229258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.229231    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca" pod "14e1decf-a43d-4327-9936-c9f34af0e317" (UID: "14e1decf-a43d-4327-9936-c9f34af0e317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:26.254811 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:26.254780    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:15:26.327822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.327791    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:26.327822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.327818    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pb6\" (UniqueName: \"kubernetes.io/projected/14e1decf-a43d-4327-9936-c9f34af0e317-kube-api-access-k9pb6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:26.328041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.327832    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e1decf-a43d-4327-9936-c9f34af0e317-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:26.883525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.883489    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:26.995447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.995417    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"
Nov 04 01:15:26.995447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.995422    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s" event={"ID":"14e1decf-a43d-4327-9936-c9f34af0e317","Type":"ContainerDied","Data":"3a595f16b10e5b714052b1432a4e39c920d8ef660a08f7b68c8d160209872823"}
Nov 04 01:15:26.995695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:26.995470    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a595f16b10e5b714052b1432a4e39c920d8ef660a08f7b68c8d160209872823"
Nov 04 01:15:27.673913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:27.673881    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:28.473926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:28.473889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:29.158660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:29.158629    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:29.506234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:29.506126    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:29.520891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:29.520865    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:29.599926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:29.599888    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:29.817059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:29.816963    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:30.163015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:30.162279    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:15:30.985677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:30.985641    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:31.558150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.558106    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:31.739040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.738982    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"]
Nov 04 01:15:31.739568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739550    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:31.739568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739565    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7932a793-995c-4f0c-9d4e-51cc775b8b54" containerName="cleanup"
Nov 04 01:15:31.739771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739588    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:31.739771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739593    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:31.739771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739601    5797 cpu_manager.go:414] "containerMap: removing container" podUID="14e1decf-a43d-4327-9936-c9f34af0e317" containerName="revoke-keys"
Nov 04 01:15:31.739771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739608    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e1decf-a43d-4327-9936-c9f34af0e317" containerName="revoke-keys"
Nov 04 01:15:31.739771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739714    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc48035-c08c-41f2-91eb-7bd56d7deba3" containerName="cleanup"
Nov 04 01:15:31.739771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.739732    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e1decf-a43d-4327-9936-c9f34af0e317" containerName="revoke-keys"
Nov 04 01:15:31.744600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.744578    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.758831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.758804    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"]
Nov 04 01:15:31.811338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811254    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwpj\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811306    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811456    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811540    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811561    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811608    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811640    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811671    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.811800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.811785    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913264    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913316    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913371    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913401    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913440    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913558    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913632    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwpj\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913681    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913739    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913763    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913870    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.913983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.913946    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.914139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.914073    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.914203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.914171    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.915980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.915951    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.916135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.916116    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.925239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.925211    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:31.925400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:31.925381    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwpj\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:32.061174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:32.061136    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:32.399723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:32.399656    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:32.413129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:32.413103    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"]
Nov 04 01:15:32.416164 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:32.416128    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00f3b41_2394_44ed_9802_cb14cb74f785.slice/crio-3525a18fc005113172ce90a69ab98aead6976fbd225eeb91046d4c5f3410276f WatchSource:0}: Error finding container 3525a18fc005113172ce90a69ab98aead6976fbd225eeb91046d4c5f3410276f: Status 404 returned error can't find the container with id 3525a18fc005113172ce90a69ab98aead6976fbd225eeb91046d4c5f3410276f
Nov 04 01:15:32.418662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:32.418623    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:32.418755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:32.418704    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:32.418826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:32.418751    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:33.010496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.010463    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"]
Nov 04 01:15:33.015985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.015961    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.023554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.023526    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:15:33.023709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.023687    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:15:33.025065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.025041    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"]
Nov 04 01:15:33.028793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.028766    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.028903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.028885    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.028960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.028933    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvcq5\" (UniqueName: \"kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.122419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.122306    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" event={"ID":"a00f3b41-2394-44ed-9802-cb14cb74f785","Type":"ContainerStarted","Data":"34f78ede5b29d54d9e73ce2985933050be0ddf4ce2203f914af4ab65b9176108"}
Nov 04 01:15:33.122419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.122379    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" event={"ID":"a00f3b41-2394-44ed-9802-cb14cb74f785","Type":"ContainerStarted","Data":"3525a18fc005113172ce90a69ab98aead6976fbd225eeb91046d4c5f3410276f"}
Nov 04 01:15:33.129956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.129914    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvcq5\" (UniqueName: \"kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.130330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.130301    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.130494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.130442    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.131168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.131135    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.133534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.133513    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.141827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.141801    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvcq5\" (UniqueName: \"kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5\") pod \"maas-api-revoke-sub-0aa44a09-fzrnt\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.173011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.169171    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podStartSLOduration=2.169146691 podStartE2EDuration="2.169146691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:33.153599881 +0000 UTC m=+720.795506133" watchObservedRunningTime="2026-11-04 01:15:33.169146691 +0000 UTC m=+720.811052941"
Nov 04 01:15:33.187442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.187411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:33.333343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.333272    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:33.676783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:33.676750    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"]
Nov 04 01:15:33.679244 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:33.679208    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0108cf1d_d038_4d3c_888a_fd49a73c8d1c.slice/crio-78f5b4ae8cbdf4c3e2b51938baa855470fe8a1e43edc3fecf534862f7d02b28b WatchSource:0}: Error finding container 78f5b4ae8cbdf4c3e2b51938baa855470fe8a1e43edc3fecf534862f7d02b28b: Status 404 returned error can't find the container with id 78f5b4ae8cbdf4c3e2b51938baa855470fe8a1e43edc3fecf534862f7d02b28b
Nov 04 01:15:34.008167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.008088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:34.074583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.074352    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:34.076547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.076224    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 04 01:15:34.076547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.076278    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 04 01:15:34.130885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.130853    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt" event={"ID":"0108cf1d-d038-4d3c-888a-fd49a73c8d1c","Type":"ContainerStarted","Data":"f8ed229bc98879386cc2aa671b6e3f4335733345e6311d1df3f1d8a0b4b184d9"}
Nov 04 01:15:34.131045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.130914    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt" event={"ID":"0108cf1d-d038-4d3c-888a-fd49a73c8d1c","Type":"ContainerStarted","Data":"78f5b4ae8cbdf4c3e2b51938baa855470fe8a1e43edc3fecf534862f7d02b28b"}
Nov 04 01:15:34.175549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.175484    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt" podStartSLOduration=2.175462347 podStartE2EDuration="2.175462347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:34.168407282 +0000 UTC m=+721.810313563" watchObservedRunningTime="2026-11-04 01:15:34.175462347 +0000 UTC m=+721.817368598"
Nov 04 01:15:34.978678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:34.978570    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:35.072418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:35.072379    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 04 01:15:35.072605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:35.072447    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 04 01:15:35.138932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:35.138896    5797 generic.go:356] "Generic (PLEG): container finished" podID="0108cf1d-d038-4d3c-888a-fd49a73c8d1c" containerID="f8ed229bc98879386cc2aa671b6e3f4335733345e6311d1df3f1d8a0b4b184d9" exitCode=0
Nov 04 01:15:35.139329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:35.138935    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt" event={"ID":"0108cf1d-d038-4d3c-888a-fd49a73c8d1c","Type":"ContainerDied","Data":"f8ed229bc98879386cc2aa671b6e3f4335733345e6311d1df3f1d8a0b4b184d9"}
Nov 04 01:15:35.696413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:35.696380    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:36.072257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.072149    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 04 01:15:36.072257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.072218    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 04 01:15:36.382529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.382498    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:36.398778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.398749    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5\" (UniqueName: \"kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5\") pod \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") "
Nov 04 01:15:36.398920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.398799    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca\") pod \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") "
Nov 04 01:15:36.399025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.399011    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca\") pod \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\" (UID: \"0108cf1d-d038-4d3c-888a-fd49a73c8d1c\") "
Nov 04 01:15:36.399381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.399346    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca" pod "0108cf1d-d038-4d3c-888a-fd49a73c8d1c" (UID: "0108cf1d-d038-4d3c-888a-fd49a73c8d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:15:36.400801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.400769    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca" pod "0108cf1d-d038-4d3c-888a-fd49a73c8d1c" (UID: "0108cf1d-d038-4d3c-888a-fd49a73c8d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:15:36.400884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.400853    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5" pod "0108cf1d-d038-4d3c-888a-fd49a73c8d1c" (UID: "0108cf1d-d038-4d3c-888a-fd49a73c8d1c"). InnerVolumeSpecName "kube-api-access-mvcq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:15:36.500433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.500396    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:36.500433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.500436    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvcq5\" (UniqueName: \"kubernetes.io/projected/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-kube-api-access-mvcq5\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:36.500653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.500452    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0108cf1d-d038-4d3c-888a-fd49a73c8d1c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:15:36.568015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:36.558957    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:37.072186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.072119    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 04 01:15:37.072353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.072185    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 04 01:15:37.181549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.181509    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt" event={"ID":"0108cf1d-d038-4d3c-888a-fd49a73c8d1c","Type":"ContainerDied","Data":"78f5b4ae8cbdf4c3e2b51938baa855470fe8a1e43edc3fecf534862f7d02b28b"}
Nov 04 01:15:37.181734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.181566    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f5b4ae8cbdf4c3e2b51938baa855470fe8a1e43edc3fecf534862f7d02b28b"
Nov 04 01:15:37.181734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.181624    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"
Nov 04 01:15:37.329094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.321569    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:37.523399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:37.523368    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:38.072015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:38.071015    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 04 01:15:38.072015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:38.071074    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 04 01:15:38.194755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:38.194725    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:38.956614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:38.956579    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:39.062070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:39.062032    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 04 01:15:39.062251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:39.062090    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 04 01:15:39.685276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:39.685199    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:40.068657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.068570    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:40.069945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.069918    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:40.071079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.071052    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:15:40.384879 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.384846    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:40.913834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.913799    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"]
Nov 04 01:15:40.914411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.914396    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0108cf1d-d038-4d3c-888a-fd49a73c8d1c" containerName="revoke-keys"
Nov 04 01:15:40.914495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.914412    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0108cf1d-d038-4d3c-888a-fd49a73c8d1c" containerName="revoke-keys"
Nov 04 01:15:40.914534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.914529    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0108cf1d-d038-4d3c-888a-fd49a73c8d1c" containerName="revoke-keys"
Nov 04 01:15:40.919487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.919462    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.925680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.925655    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 04 01:15:40.925789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.925709    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 04 01:15:40.925969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.925950    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 04 01:15:40.926104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.925985    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mdkcc\""
Nov 04 01:15:40.930566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.930547    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"]
Nov 04 01:15:40.957594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.957564    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.957732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.957669    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.957787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.957775    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.957845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.957798    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fwx6\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.957845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.957818    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.958024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.957978    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:40.958101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:40.958040    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059459    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059500    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwx6\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059528    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059615    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059638    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059703    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.059927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.059794    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.060822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.060771    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.062053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.062032    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.062217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.062182    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.062348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.062329    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.062558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.062544    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.074053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.074031    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.074384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.074359    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwx6\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r7v8r\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.177117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.177029    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:41.234500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.234472    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:41.519406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.519375    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"]
Nov 04 01:15:41.519791 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:41.519762    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e7cd6f_8996_4d26_956b_453701b30f64.slice/crio-33dd09cc1325f0d7cd7d58bf03e50eb36aa292aa8d7daacf12c3653ab134d766 WatchSource:0}: Error finding container 33dd09cc1325f0d7cd7d58bf03e50eb36aa292aa8d7daacf12c3653ab134d766: Status 404 returned error can't find the container with id 33dd09cc1325f0d7cd7d58bf03e50eb36aa292aa8d7daacf12c3653ab134d766
Nov 04 01:15:41.934814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:41.934782    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:42.234557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:42.234467    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" event={"ID":"67e7cd6f-8996-4d26-956b-453701b30f64","Type":"ContainerStarted","Data":"33dd09cc1325f0d7cd7d58bf03e50eb36aa292aa8d7daacf12c3653ab134d766"}
Nov 04 01:15:42.727785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:42.727717    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:42.936863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:42.936817    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:43.598912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:43.598860    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:44.310873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:44.310801    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:44.982464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:44.982427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:45.694270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:45.694236    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:46.580229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:46.580158    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:47.210346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.210296    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:47.600712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.600622    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"]
Nov 04 01:15:47.704585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.704557    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"]
Nov 04 01:15:47.704822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.704808    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"]
Nov 04 01:15:47.710941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.709983    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.714980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.714939    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1036e8-openshift-default-dockercfg-wbg6m\""
Nov 04 01:15:47.723183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.723149    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"]
Nov 04 01:15:47.723471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.723456    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.782784    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.782853    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.782890    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.782920    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.782961    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783004    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783030    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783055    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783094    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783210    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783235    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783258    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783305    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8s4\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783328    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783351    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783380    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783404    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.783792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.783451    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4qv\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.887247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.887201    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.887525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.887504    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.887668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.887651    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.887847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.887813    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8s4\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.889359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889333    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.889512 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889344    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.889635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889617    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.889753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889737    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.889866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889849    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.890017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889984    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4qv\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.890200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890176    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.890348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890326    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.890486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890471    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.890601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890587    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.890733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890718    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.890833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890819    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.890932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.890920    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.897073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.897045    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.897259 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.897242    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.901807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.901773    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.902425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.902388    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.902866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.902834    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.903763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.903732    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.904379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889779    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.904570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.889248    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.922273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.913918    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.922273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.915385    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:47.932808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.932376    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.935590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.933593    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8s4\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.935590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.934807    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.936244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.936214    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.936521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.936463    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:47.936644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.936567    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.940021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.937413    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.972572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.971732    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4qv\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:47.982668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:47.982599    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:48.001116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.000003    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:48.100658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.100614    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:48.112105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.107460    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:48.471173 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:15:48.471139    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7f34d0_b8a1_418f_895b_3a5b61c57bda.slice/crio-a25a35ea866b953ee1f123866d88357746e1164b802b518b37e7ec450ecdae73 WatchSource:0}: Error finding container a25a35ea866b953ee1f123866d88357746e1164b802b518b37e7ec450ecdae73: Status 404 returned error can't find the container with id a25a35ea866b953ee1f123866d88357746e1164b802b518b37e7ec450ecdae73
Nov 04 01:15:48.471776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.471735    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"]
Nov 04 01:15:48.481562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.480092    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:48.481562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.480282    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:48.481562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.480412    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:48.482245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.482203    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:48.482383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.482355    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:48.482510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.482483    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:48.504780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.504714    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"]
Nov 04 01:15:48.627030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.626847    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" event={"ID":"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097","Type":"ContainerStarted","Data":"b95a5c2de698e523d23312d3e55b13aced1d30bb36e7a63336b0a8717d109b90"}
Nov 04 01:15:48.631351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.629661    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" event={"ID":"9c7f34d0-b8a1-418f-895b-3a5b61c57bda","Type":"ContainerStarted","Data":"a25a35ea866b953ee1f123866d88357746e1164b802b518b37e7ec450ecdae73"}
Nov 04 01:15:48.975013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:48.974952    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:49.638473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:49.638435    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" event={"ID":"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097","Type":"ContainerStarted","Data":"9174a9d3e91e722c829ca91a53bd9c3e5300363fe24dbd12001ec72623e26528"}
Nov 04 01:15:49.640204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:49.640178    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" event={"ID":"9c7f34d0-b8a1-418f-895b-3a5b61c57bda","Type":"ContainerStarted","Data":"754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1"}
Nov 04 01:15:49.705750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:49.705684    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podStartSLOduration=2.705636344 podStartE2EDuration="2.705636344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:49.704630727 +0000 UTC m=+737.346536974" watchObservedRunningTime="2026-11-04 01:15:49.705636344 +0000 UTC m=+737.347542594"
Nov 04 01:15:49.707960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:49.707909    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podStartSLOduration=2.707899393 podStartE2EDuration="2.707899393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:49.672325685 +0000 UTC m=+737.314231955" watchObservedRunningTime="2026-11-04 01:15:49.707899393 +0000 UTC m=+737.349805647"
Nov 04 01:15:49.837872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:49.837839    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:50.121014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.114682    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:50.121014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.114812    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 04 01:15:50.121014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.114856    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 04 01:15:50.121014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.116420    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:50.121014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.116544    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:50.121014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.116582    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:50.837966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:50.837924    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:51.101324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:51.101234    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 04 01:15:51.101547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:51.101520    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 04 01:15:51.108194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:51.108167    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:51.108327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:51.108214    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:51.802010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:51.801665    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:52.101411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:52.101305    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 04 01:15:52.101411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:52.101365    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 04 01:15:52.108015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:52.107913    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:52.108015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:52.107957    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:52.978012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:52.969563    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:53.102155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:53.101366    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 04 01:15:53.102155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:53.101429    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 04 01:15:53.108197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:53.108169    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:53.108387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:53.108211    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:53.930211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:53.930124    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:54.101534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:54.101492    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 04 01:15:54.101823 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:54.101790    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 04 01:15:54.108372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:54.108347    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:54.108889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:54.108863    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:54.839970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:54.839933    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:55.101770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.101312    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 04 01:15:55.101770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.101373    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 04 01:15:55.108164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.108136    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:55.108309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.108187    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:55.662683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.662645    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:55.906242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.906201    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:55.906361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:55.906290    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:15:56.108311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.108203    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 04 01:15:56.108311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.108266    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 04 01:15:56.280941 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:15:56.280849    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:15:56.340472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.340448    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:56.341836 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.341392    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:56.342731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.342708    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:15:56.442211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.441930    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:56.752505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.752434    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" event={"ID":"67e7cd6f-8996-4d26-956b-453701b30f64","Type":"ContainerStarted","Data":"6b122cb32add62c3374fb07da786e2bd874194bfac7e94d39cf08f825cf4db48"}
Nov 04 01:15:56.781858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:56.781795    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" podStartSLOduration=2.398784234 podStartE2EDuration="16.781777649s" totalImagesPullingTime="14.382993415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:56.779149249 +0000 UTC m=+744.421055500" watchObservedRunningTime="2026-11-04 01:15:56.781777649 +0000 UTC m=+744.423683899"
Nov 04 01:15:57.114402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.114375    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:57.114898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.114883    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:57.115585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.115563    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:15:57.175104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.175067    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:57.629829 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.629783    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:57.760615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.760581    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:57.761911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:57.761881    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:15:58.048215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:58.048132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:58.890587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:58.890547    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:59.706446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.706406    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"]
Nov 04 01:15:59.716625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.716593    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.724665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.724627    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:15:59.724832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.724681    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:15:59.735706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.735672    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:15:59.738636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.738605    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"]
Nov 04 01:15:59.792463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.792427    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kxm\" (UniqueName: \"kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.792620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.792479    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.792667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.792620    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.893577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.893539    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.894064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.893704    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.894064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.893957    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kxm\" (UniqueName: \"kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.894391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.894370    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.896101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.896081    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:15:59.906197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:15:59.906167    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kxm\" (UniqueName: \"kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm\") pod \"maas-api-revoke-sub-f61f985e-l6l8b\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:16:00.044082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.043973    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:16:00.328334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.328308    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"]
Nov 04 01:16:00.330518 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:00.330487    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44ce06f_4dca_4e98_bfe7_f29944d8dcea.slice/crio-1882ce8c4b59194a684cd759eab4242c36ccb1125467c10a24fca2e5c5bf001d WatchSource:0}: Error finding container 1882ce8c4b59194a684cd759eab4242c36ccb1125467c10a24fca2e5c5bf001d: Status 404 returned error can't find the container with id 1882ce8c4b59194a684cd759eab4242c36ccb1125467c10a24fca2e5c5bf001d
Nov 04 01:16:00.480900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.480868    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:00.678487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.678453    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:00.781303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.781269    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b" event={"ID":"a44ce06f-4dca-4e98-bfe7-f29944d8dcea","Type":"ContainerStarted","Data":"fcf2cf02fb799c670b0ed5ce89930ab6d170d3977f6ac0623702a7651585c4d1"}
Nov 04 01:16:00.781433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.781336    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b" event={"ID":"a44ce06f-4dca-4e98-bfe7-f29944d8dcea","Type":"ContainerStarted","Data":"1882ce8c4b59194a684cd759eab4242c36ccb1125467c10a24fca2e5c5bf001d"}
Nov 04 01:16:00.801401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:00.801357    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b" podStartSLOduration=1.801344055 podStartE2EDuration="1.801344055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:00.798421134 +0000 UTC m=+748.440327384" watchObservedRunningTime="2026-11-04 01:16:00.801344055 +0000 UTC m=+748.443250305"
Nov 04 01:16:01.204608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:01.204565    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:01.790685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:01.790651    5797 generic.go:356] "Generic (PLEG): container finished" podID="a44ce06f-4dca-4e98-bfe7-f29944d8dcea" containerID="fcf2cf02fb799c670b0ed5ce89930ab6d170d3977f6ac0623702a7651585c4d1" exitCode=0
Nov 04 01:16:01.790870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:01.790741    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b" event={"ID":"a44ce06f-4dca-4e98-bfe7-f29944d8dcea","Type":"ContainerDied","Data":"fcf2cf02fb799c670b0ed5ce89930ab6d170d3977f6ac0623702a7651585c4d1"}
Nov 04 01:16:01.951009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:01.950582    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"]
Nov 04 01:16:01.955654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:01.955627    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:01.972700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:01.972667    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"]
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.025627    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.026697    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.026736    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92n6\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.026798    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.026822    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.026946    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.027005    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.027112    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.027177    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.028550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.027228    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150504    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150583    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150679    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150736    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150788    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150873    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.150906    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92n6\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.154809    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.155028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.154863    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.156340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.156307    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.156467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.156454    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.156600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.156575    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.158346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.156937    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.158346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.157079    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.159378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.159349    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.159606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.159582    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.169048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.167539    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92n6\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.169679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.169650    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.286489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.286446    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:02.457928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.457841    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"]
Nov 04 01:16:02.643132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.631359    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.654466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.653033    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"]
Nov 04 01:16:02.654466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.653084    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"]
Nov 04 01:16:02.659459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.659425    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"]
Nov 04 01:16:02.659610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.659543    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.755062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.734835    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:02.755062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.734926    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:02.755062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.734969    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:02.755062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.739244    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"]
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850169    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850220    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850265    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850298    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850368    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850396    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850447    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850477    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74r8b\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850504    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850528    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850554    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850582    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850610    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850739    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850765    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850789    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850910    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzvrw\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.851010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.850935    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.955860    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.955918    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956019    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956051    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956109    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956142    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74r8b\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956188    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956212    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956239    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956269    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956295    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956432    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956456    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956482    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956585    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzvrw\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956609    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956670    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.970074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.956695    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.972320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.972289    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.972613    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.972782    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.973078    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.973466    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.973668    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.978096    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.978755    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:02.988390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.987791    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:02.989864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.989296    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:03.003112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:02.999702    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74r8b\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:03.004941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.004879    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:03.005898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.005818    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:03.025195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.022089    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzvrw\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:03.025195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.022376    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:03.035647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.035553    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:03.035647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.035551    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:03.103625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.103593    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:03.134037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.134005    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:03.228370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.226634    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" event={"ID":"a6b7fa46-feb6-4b67-895a-9624c2511f00","Type":"ContainerStarted","Data":"a9b1eba6ea518434a7f2e046e8e91c0303f0ef60cb3b605670f005736c486310"}
Nov 04 01:16:03.335477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.332770    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:03.951014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.950705    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:04.033882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:03.978906    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:16:04.433822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.433770    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" event={"ID":"a6b7fa46-feb6-4b67-895a-9624c2511f00","Type":"ContainerStarted","Data":"dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503"}
Nov 04 01:16:04.624304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.620094    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:04.624304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.620259    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:04.624304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.620373    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:04.636060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.636009    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podStartSLOduration=3.635972617 podStartE2EDuration="3.635972617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:04.608108266 +0000 UTC m=+752.250014518" watchObservedRunningTime="2026-11-04 01:16:04.635972617 +0000 UTC m=+752.277878869"
Nov 04 01:16:04.640310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.640282    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"]
Nov 04 01:16:04.640773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.640753    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:16:04.709448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.709408    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca\") pod \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") "
Nov 04 01:16:04.709757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.709742    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm\" (UniqueName: \"kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm\") pod \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") "
Nov 04 01:16:04.709885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.709873    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca\") pod \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\" (UID: \"a44ce06f-4dca-4e98-bfe7-f29944d8dcea\") "
Nov 04 01:16:04.713086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.713064    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca" pod "a44ce06f-4dca-4e98-bfe7-f29944d8dcea" (UID: "a44ce06f-4dca-4e98-bfe7-f29944d8dcea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:04.713470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.713454    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca" pod "a44ce06f-4dca-4e98-bfe7-f29944d8dcea" (UID: "a44ce06f-4dca-4e98-bfe7-f29944d8dcea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:04.716176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.716153    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm" pod "a44ce06f-4dca-4e98-bfe7-f29944d8dcea" (UID: "a44ce06f-4dca-4e98-bfe7-f29944d8dcea"). InnerVolumeSpecName "kube-api-access-c5kxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:04.735480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.735446    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"]
Nov 04 01:16:04.741915 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:04.741884    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07da5e23_0dfd_4c32_80b2_3b730e9bce8b.slice/crio-2f883f8e7f157be7d04552c7d43bfc354018e163fcd5cc7744d7acc23bb498f8 WatchSource:0}: Error finding container 2f883f8e7f157be7d04552c7d43bfc354018e163fcd5cc7744d7acc23bb498f8: Status 404 returned error can't find the container with id 2f883f8e7f157be7d04552c7d43bfc354018e163fcd5cc7744d7acc23bb498f8
Nov 04 01:16:04.773015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.747884    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:04.815880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.799758    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:04.815880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.799910    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:04.815880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.811735    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:04.815880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.811767    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:04.815880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.811912    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5kxm\" (UniqueName: \"kubernetes.io/projected/a44ce06f-4dca-4e98-bfe7-f29944d8dcea-kube-api-access-c5kxm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:04.922659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:04.922582    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:05.290622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.289340    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:05.290622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.289569    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:05.290622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.289700    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:05.290622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.290332    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:05.444436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.444352    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"
Nov 04 01:16:05.444436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.444365    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b" event={"ID":"a44ce06f-4dca-4e98-bfe7-f29944d8dcea","Type":"ContainerDied","Data":"1882ce8c4b59194a684cd759eab4242c36ccb1125467c10a24fca2e5c5bf001d"}
Nov 04 01:16:05.445103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.444438    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1882ce8c4b59194a684cd759eab4242c36ccb1125467c10a24fca2e5c5bf001d"
Nov 04 01:16:05.454178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.454141    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" event={"ID":"07da5e23-0dfd-4c32-80b2-3b730e9bce8b","Type":"ContainerStarted","Data":"2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e"}
Nov 04 01:16:05.454439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.454415    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" event={"ID":"07da5e23-0dfd-4c32-80b2-3b730e9bce8b","Type":"ContainerStarted","Data":"2f883f8e7f157be7d04552c7d43bfc354018e163fcd5cc7744d7acc23bb498f8"}
Nov 04 01:16:05.488509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.486982    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podStartSLOduration=3.486963637 podStartE2EDuration="3.486963637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:05.484952739 +0000 UTC m=+753.126859009" watchObservedRunningTime="2026-11-04 01:16:05.486963637 +0000 UTC m=+753.128869885"
Nov 04 01:16:05.489298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.489273    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" event={"ID":"8f544061-98e2-48f2-a6a2-b6d317169a74","Type":"ContainerStarted","Data":"f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5"}
Nov 04 01:16:05.489647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.489623    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" event={"ID":"8f544061-98e2-48f2-a6a2-b6d317169a74","Type":"ContainerStarted","Data":"473848d1309de03df1b9153c2b75ac0a9e30c5285b39fdd7246cc1c1e4a99f63"}
Nov 04 01:16:05.517276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:05.517211    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podStartSLOduration=3.517189646 podStartE2EDuration="3.517189646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:05.516110889 +0000 UTC m=+753.158017164" watchObservedRunningTime="2026-11-04 01:16:05.517189646 +0000 UTC m=+753.159095896"
Nov 04 01:16:06.136391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.136358    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:06.189016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.188134    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:06.189016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.188200    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:06.308191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.304826    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:06.308191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.304901    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:06.340785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.340747    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:06.342150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.342122    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:06.345014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.342358    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:06.543073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:06.541677    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:07.136538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.136501    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:07.136824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.136791    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:07.292013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.289486    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:07.292013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.289561    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:07.338421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.338384    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:07.338677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.338647    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:07.342011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.341051    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"]
Nov 04 01:16:07.342011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.341632    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a44ce06f-4dca-4e98-bfe7-f29944d8dcea" containerName="revoke-keys"
Nov 04 01:16:07.342011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.341643    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44ce06f-4dca-4e98-bfe7-f29944d8dcea" containerName="revoke-keys"
Nov 04 01:16:07.342011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.341776    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44ce06f-4dca-4e98-bfe7-f29944d8dcea" containerName="revoke-keys"
Nov 04 01:16:07.348011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.347584    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"]
Nov 04 01:16:07.348011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.347693    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.425069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.422144    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:16:07.428355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.428312    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:16:07.460648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.447667    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxtg\" (UniqueName: \"kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.466011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.460847    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.466011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.460933    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.565016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.564256    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxtg\" (UniqueName: \"kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.565016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.564345    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.565016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.564386    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.565677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.565411    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.620054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.613129    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxtg\" (UniqueName: \"kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.620766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.619576    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1cae41-5cdrw\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:07.633727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.628516    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:07.785062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:07.781774    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:08.138159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.138117    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:08.138341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.138205    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:08.337347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.335037    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:08.337347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.335089    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:08.337347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.336042    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:08.337347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.336082    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:08.341817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.341370    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"]
Nov 04 01:16:08.837332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.837293    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw" event={"ID":"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3","Type":"ContainerStarted","Data":"c79c865a7a48fa6eecd04e497e321b29d042e70543ad752268ad6567f9086d41"}
Nov 04 01:16:08.996018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:08.995634    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:09.135542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.135502    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:09.135825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.135792    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:09.294414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.294318    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:09.294660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.294626    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:09.336909 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.336872    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:09.337162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.337142    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:09.846235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.846149    5797 generic.go:356] "Generic (PLEG): container finished" podID="eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" containerID="69e837bc1683ed2ea071a07beb61ea66c71261dbec47f848508076f19b5629d6" exitCode=0
Nov 04 01:16:09.846706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:09.846234    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw" event={"ID":"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3","Type":"ContainerDied","Data":"69e837bc1683ed2ea071a07beb61ea66c71261dbec47f848508076f19b5629d6"}
Nov 04 01:16:10.135458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.135413    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:10.135734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.135706    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:10.221862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.221743    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:10.291532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.291487    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:10.291839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.291789    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:10.336457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.336422    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:10.336688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:10.336670    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:11.137043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.135773    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:11.137043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.135833    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:11.315679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.314536    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:16:11.315679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.314585    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:16:11.328009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.322211    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:11.337287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.337246    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:11.337486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.337312    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:11.390294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.365442    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:11.432337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.432303    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca\") pod \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") "
Nov 04 01:16:11.434888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.434857    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca\") pod \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") "
Nov 04 01:16:11.435207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.435184    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg\" (UniqueName: \"kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg\") pod \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\" (UID: \"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3\") "
Nov 04 01:16:11.438812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.438789    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg" pod "eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" (UID: "eb14c7b9-b8d3-41f4-aac8-9e79d27472b3"). InnerVolumeSpecName "kube-api-access-2pxtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:11.439231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.439213    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca" pod "eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" (UID: "eb14c7b9-b8d3-41f4-aac8-9e79d27472b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:11.476632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.476577    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca" pod "eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" (UID: "eb14c7b9-b8d3-41f4-aac8-9e79d27472b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:11.536906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.536872    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:11.536906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.536901    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:11.536906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.536911    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pxtg\" (UniqueName: \"kubernetes.io/projected/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3-kube-api-access-2pxtg\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:11.883358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.883326    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"
Nov 04 01:16:11.883550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.883327    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw" event={"ID":"eb14c7b9-b8d3-41f4-aac8-9e79d27472b3","Type":"ContainerDied","Data":"c79c865a7a48fa6eecd04e497e321b29d042e70543ad752268ad6567f9086d41"}
Nov 04 01:16:11.883550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:11.883486    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79c865a7a48fa6eecd04e497e321b29d042e70543ad752268ad6567f9086d41"
Nov 04 01:16:12.135524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.135420    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 04 01:16:12.135524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.135473    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 04 01:16:12.290174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.290103    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:12.301097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.301068    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:12.301625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.301605    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:12.302489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.302473    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:16:12.336342 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.336306    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 04 01:16:12.336509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:12.336369    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 04 01:16:13.144180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.144149    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:13.144685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.144664    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:13.146095 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.146071    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:16:13.182079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.181362    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:13.350131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.350105    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:13.350648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.350592    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:13.351362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.351345    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:16:13.987725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:13.987695    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:14.722243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:14.722193    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:15.476180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:15.476139    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:16.185137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.185099    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"]
Nov 04 01:16:16.185713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.185699    5797 cpu_manager.go:414] "containerMap: removing container" podUID="eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" containerName="revoke-keys"
Nov 04 01:16:16.185763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.185714    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" containerName="revoke-keys"
Nov 04 01:16:16.185845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.185836    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" containerName="revoke-keys"
Nov 04 01:16:16.189295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.189272    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.196172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.196148    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:16:16.196309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.196278    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:16:16.202412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.202385    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"]
Nov 04 01:16:16.276922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.276886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:16.309201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.309170    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.309407 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.309231    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfgm\" (UniqueName: \"kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.309407 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.309378    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.410944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.410854    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.411108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.411088    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.411154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.411123    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfgm\" (UniqueName: \"kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.411786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.411767    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.413235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.413206    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.424281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.424251    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfgm\" (UniqueName: \"kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm\") pod \"maas-api-revoke-sub-3bc7868d-28twq\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.505335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.505304    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:16.787978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.787946    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"]
Nov 04 01:16:16.788537 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:16.788513    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ce2c49_d221_41ec_a8a3_c4acaa95cb71.slice/crio-bf7e5ef5d38af70f85b0cc069036ca954a00b0e118832bebe1814ee129712102 WatchSource:0}: Error finding container bf7e5ef5d38af70f85b0cc069036ca954a00b0e118832bebe1814ee129712102: Status 404 returned error can't find the container with id bf7e5ef5d38af70f85b0cc069036ca954a00b0e118832bebe1814ee129712102
Nov 04 01:16:16.920353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:16.920320    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq" event={"ID":"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71","Type":"ContainerStarted","Data":"bf7e5ef5d38af70f85b0cc069036ca954a00b0e118832bebe1814ee129712102"}
Nov 04 01:16:17.014969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.014037    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"]
Nov 04 01:16:17.034901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.034861    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:17.044501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.044424    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"]
Nov 04 01:16:17.044664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.044574    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233275    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233314    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233415    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233463    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233527    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233617    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233696    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5465\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233757    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.233975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.233792    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.335887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.335787    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5465\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.335887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.335870    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.335916    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336050    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336076    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336137    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336201    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336243    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336375    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336704    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336765    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.336913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.336887    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.337857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.337174    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.337953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.337295    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.339524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.339336    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.339524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.339497    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.357038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.355208    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"]
Nov 04 01:16:17.357038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.356007    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5465\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.357038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.356930    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token\") pod \"e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") " pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.362359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.362206    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.373331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.373305    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:17.374444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.374419    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"]
Nov 04 01:16:17.539631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.539594    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.539631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.539633    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.539861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.539804    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.539861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.539851    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.539967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.539880    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.540049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.539965    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.540049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.540030    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.540148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.540076    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.540203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.540147    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kgt\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641786    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641841    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641864    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641892    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641919    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641961    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.641978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.641978    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kgt\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642114    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642142    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642317    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642380    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642451    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642684    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.642760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.642744    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.644243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.644220    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.644476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.644459    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.653683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.653660    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.654076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.654053    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kgt\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.687923 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:17.687893    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7af977b_c72f_4da5_81b1_e5eb0de7ead0.slice/crio-c176b920f0625091d41954bfc242d549685e3e10084f9f0f414b2356a989e31f WatchSource:0}: Error finding container c176b920f0625091d41954bfc242d549685e3e10084f9f0f414b2356a989e31f: Status 404 returned error can't find the container with id c176b920f0625091d41954bfc242d549685e3e10084f9f0f414b2356a989e31f
Nov 04 01:16:17.690215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.690180    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"]
Nov 04 01:16:17.690737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.690707    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:17.690786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.690773    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:17.690830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.690802    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:17.697058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.695627    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:17.847775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.847621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:17.935047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.933082    5797 generic.go:356] "Generic (PLEG): container finished" podID="a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" containerID="9ad2d4360864c677cb84d54342534ddfd9266a8a47ece63b3bb085f7e336d5c9" exitCode=0
Nov 04 01:16:17.935047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.933162    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq" event={"ID":"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71","Type":"ContainerDied","Data":"9ad2d4360864c677cb84d54342534ddfd9266a8a47ece63b3bb085f7e336d5c9"}
Nov 04 01:16:17.935363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.935341    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" event={"ID":"e7af977b-c72f-4da5-81b1-e5eb0de7ead0","Type":"ContainerStarted","Data":"e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af"}
Nov 04 01:16:17.935425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.935386    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" event={"ID":"e7af977b-c72f-4da5-81b1-e5eb0de7ead0","Type":"ContainerStarted","Data":"c176b920f0625091d41954bfc242d549685e3e10084f9f0f414b2356a989e31f"}
Nov 04 01:16:17.984509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.984446    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podStartSLOduration=1.984428439 podStartE2EDuration="1.984428439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:17.983395235 +0000 UTC m=+765.625301547" watchObservedRunningTime="2026-11-04 01:16:17.984428439 +0000 UTC m=+765.626334690"
Nov 04 01:16:17.995888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.995856    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"]
Nov 04 01:16:17.996419 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:17.996388    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da19693_73e3_4bb8_9637_c68ec381334e.slice/crio-7dc3de28bb1d482c5398218a974cbbc8fc943f60ae3188a4fbdb5ac737fd2db5 WatchSource:0}: Error finding container 7dc3de28bb1d482c5398218a974cbbc8fc943f60ae3188a4fbdb5ac737fd2db5: Status 404 returned error can't find the container with id 7dc3de28bb1d482c5398218a974cbbc8fc943f60ae3188a4fbdb5ac737fd2db5
Nov 04 01:16:17.999751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.999722    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:17.999821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.999785    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:17.999869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:17.999816    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:18.377016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.375344    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:18.377016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.375477    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:18.377016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.375521    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:18.744781 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.744699    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:18.945345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.945278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:18.991317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.991280    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" event={"ID":"0da19693-73e3-4bb8-9637-c68ec381334e","Type":"ContainerStarted","Data":"b3e2aa22ac64b672ec3f91e951b78ea849992351666acd99b570c7f81f4d10d7"}
Nov 04 01:16:18.997293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:18.993146    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" event={"ID":"0da19693-73e3-4bb8-9637-c68ec381334e","Type":"ContainerStarted","Data":"7dc3de28bb1d482c5398218a974cbbc8fc943f60ae3188a4fbdb5ac737fd2db5"}
Nov 04 01:16:19.027012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.026924    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podStartSLOduration=2.026894535 podStartE2EDuration="2.026894535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:19.025023635 +0000 UTC m=+766.666929918" watchObservedRunningTime="2026-11-04 01:16:19.026894535 +0000 UTC m=+766.668800785"
Nov 04 01:16:19.346316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.346286    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:19.405029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.403652    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:19.405029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.403709    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:19.514896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.514864    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm\" (UniqueName: \"kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm\") pod \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") "
Nov 04 01:16:19.515097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.515015    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca\") pod \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") "
Nov 04 01:16:19.515831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.515148    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca\") pod \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\" (UID: \"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71\") "
Nov 04 01:16:19.515953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.515863    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca" pod "a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" (UID: "a7ce2c49-d221-41ec-a8a3-c4acaa95cb71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:19.517586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.517558    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca" pod "a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" (UID: "a7ce2c49-d221-41ec-a8a3-c4acaa95cb71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:19.517683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.517589    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm" pod "a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" (UID: "a7ce2c49-d221-41ec-a8a3-c4acaa95cb71"). InnerVolumeSpecName "kube-api-access-2cfgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:19.617728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.617695    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:19.617728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.617723    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfgm\" (UniqueName: \"kubernetes.io/projected/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-kube-api-access-2cfgm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:19.617901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.617735    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:19.697858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.697757    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:19.703528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.703454    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:16:19.703528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.703521    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:16:19.986071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:19.977431    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:20.030012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.026258    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"
Nov 04 01:16:20.030012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.028595    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq" event={"ID":"a7ce2c49-d221-41ec-a8a3-c4acaa95cb71","Type":"ContainerDied","Data":"bf7e5ef5d38af70f85b0cc069036ca954a00b0e118832bebe1814ee129712102"}
Nov 04 01:16:20.030012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.028651    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7e5ef5d38af70f85b0cc069036ca954a00b0e118832bebe1814ee129712102"
Nov 04 01:16:20.396016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.395274    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:20.396016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.395334    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:20.697555 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.697461    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:16:20.698143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:20.698109    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:16:21.020630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:21.020527    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:21.389724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:21.389682    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:21.390012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:21.389961    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:21.698501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:21.698407    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:16:21.699093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:21.699056    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:16:21.944033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:21.943750    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:22.387015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.385104    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:22.387015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.385170    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:22.442483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.442448    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"]
Nov 04 01:16:22.443124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.443108    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" containerName="revoke-keys"
Nov 04 01:16:22.443124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.443125    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" containerName="revoke-keys"
Nov 04 01:16:22.443254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.443215    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" containerName="revoke-keys"
Nov 04 01:16:22.451014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.447147    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.475013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.474289    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:16:22.521053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.511784    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fvm\" (UniqueName: \"kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.521053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.511904    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.521053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.511943    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.524286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.524256    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"]
Nov 04 01:16:22.585865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.579141    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:16:22.619692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.619661    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fvm\" (UniqueName: \"kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.619870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.619739    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.619870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.619767    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.620532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.620513    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.622371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.622347    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.632437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.632411    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fvm\" (UniqueName: \"kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm\") pod \"maas-api-revoke-sub-4fcccd6f-vjbml\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:22.702853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.699628    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:16:22.702853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.699686    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:16:22.830403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:22.830374    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:23.216018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.213731    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:23.223378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.223342    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"]
Nov 04 01:16:23.230533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.230505    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.231429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.231059    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"]
Nov 04 01:16:23.237135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.236937    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"]
Nov 04 01:16:23.334059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.334019    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.334195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.334088    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bqh\" (UniqueName: \"kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.334276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.334264    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.386046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.380381    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:23.386046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.380439    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:23.435782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.435752    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.436065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.436033    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bqh\" (UniqueName: \"kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.436362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.436342    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.436587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.436565    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.438743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.438723    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.465756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.465724    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bqh\" (UniqueName: \"kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh\") pod \"maas-api-revoke-sub-50049503-rgfp2\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.611788 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.611757    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:23.701010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.700419    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:16:23.701010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.700486    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:16:23.944698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:23.944664    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"]
Nov 04 01:16:23.945817 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:23.945769    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9be1452_095c_4dfc_9862_49ef80ac8349.slice/crio-408228046ecc440c5f8b266940328ab1cb3420349fb063013bd2de3341527c6e WatchSource:0}: Error finding container 408228046ecc440c5f8b266940328ab1cb3420349fb063013bd2de3341527c6e: Status 404 returned error can't find the container with id 408228046ecc440c5f8b266940328ab1cb3420349fb063013bd2de3341527c6e
Nov 04 01:16:24.117635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.117540    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:24.231946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.231909    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2" event={"ID":"f9be1452-095c-4dfc-9862-49ef80ac8349","Type":"ContainerStarted","Data":"408228046ecc440c5f8b266940328ab1cb3420349fb063013bd2de3341527c6e"}
Nov 04 01:16:24.233973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.233945    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml" event={"ID":"b127ba27-cb9c-472b-ba77-c227b5962076","Type":"ContainerStarted","Data":"94815c2c789983c778f9abd229c5af4dbd56142d74e74c06d396009bc9c9a75f"}
Nov 04 01:16:24.234145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.234032    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml" event={"ID":"b127ba27-cb9c-472b-ba77-c227b5962076","Type":"ContainerStarted","Data":"3d751d5db67613631fc2f0e4d0f653a7aceef5915d5ca4a4758afd02a9cf2716"}
Nov 04 01:16:24.270256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.270210    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml" podStartSLOduration=2.270196892 podStartE2EDuration="2.270196892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:24.267194317 +0000 UTC m=+771.909100572" watchObservedRunningTime="2026-11-04 01:16:24.270196892 +0000 UTC m=+771.912103136"
Nov 04 01:16:24.374596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.374519    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:16:24.374596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.374569    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:16:24.696893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.696809    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:16:24.696893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:24.696876    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:16:25.099204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.099099    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:25.248540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.248498    5797 generic.go:356] "Generic (PLEG): container finished" podID="b127ba27-cb9c-472b-ba77-c227b5962076" containerID="94815c2c789983c778f9abd229c5af4dbd56142d74e74c06d396009bc9c9a75f" exitCode=0
Nov 04 01:16:25.248772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.248736    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml" event={"ID":"b127ba27-cb9c-472b-ba77-c227b5962076","Type":"ContainerDied","Data":"94815c2c789983c778f9abd229c5af4dbd56142d74e74c06d396009bc9c9a75f"}
Nov 04 01:16:25.251102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.250972    5797 generic.go:356] "Generic (PLEG): container finished" podID="f9be1452-095c-4dfc-9862-49ef80ac8349" containerID="2bff4fc5c2842c836d8194ecdc26a6cc4c3bf4fd6a216efe5d85f3088fe8ceb4" exitCode=0
Nov 04 01:16:25.251102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.251028    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2" event={"ID":"f9be1452-095c-4dfc-9862-49ef80ac8349","Type":"ContainerDied","Data":"2bff4fc5c2842c836d8194ecdc26a6cc4c3bf4fd6a216efe5d85f3088fe8ceb4"}
Nov 04 01:16:25.498222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.498169    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:25.498762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.498583    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:25.502065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.502010    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:16:25.734765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.734736    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:25.735218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.735193    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:25.736084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.736067    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:16:25.984065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:25.984023    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:26.254581 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:16:26.254483    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:16:26.470724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.470695    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:26.477258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.477230    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:26.512080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.511938    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca\") pod \"b127ba27-cb9c-472b-ba77-c227b5962076\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") "
Nov 04 01:16:26.512080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.512046    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm\" (UniqueName: \"kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm\") pod \"b127ba27-cb9c-472b-ba77-c227b5962076\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") "
Nov 04 01:16:26.512342 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.512224    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca\") pod \"b127ba27-cb9c-472b-ba77-c227b5962076\" (UID: \"b127ba27-cb9c-472b-ba77-c227b5962076\") "
Nov 04 01:16:26.515374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.513251    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca" pod "b127ba27-cb9c-472b-ba77-c227b5962076" (UID: "b127ba27-cb9c-472b-ba77-c227b5962076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:26.517834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.517799    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca" pod "b127ba27-cb9c-472b-ba77-c227b5962076" (UID: "b127ba27-cb9c-472b-ba77-c227b5962076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:26.519141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.519116    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm" pod "b127ba27-cb9c-472b-ba77-c227b5962076" (UID: "b127ba27-cb9c-472b-ba77-c227b5962076"). InnerVolumeSpecName "kube-api-access-66fvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:26.613614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.613584    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh\" (UniqueName: \"kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh\") pod \"f9be1452-095c-4dfc-9862-49ef80ac8349\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") "
Nov 04 01:16:26.613775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.613674    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca\") pod \"f9be1452-095c-4dfc-9862-49ef80ac8349\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") "
Nov 04 01:16:26.613858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.613817    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca\") pod \"f9be1452-095c-4dfc-9862-49ef80ac8349\" (UID: \"f9be1452-095c-4dfc-9862-49ef80ac8349\") "
Nov 04 01:16:26.614207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.614173    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca" pod "f9be1452-095c-4dfc-9862-49ef80ac8349" (UID: "f9be1452-095c-4dfc-9862-49ef80ac8349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:26.614364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.614347    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:26.614435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.614370    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:26.614435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.614387    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b127ba27-cb9c-472b-ba77-c227b5962076-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:26.614435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.614402    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fvm\" (UniqueName: \"kubernetes.io/projected/b127ba27-cb9c-472b-ba77-c227b5962076-kube-api-access-66fvm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:26.615720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.615698    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca" pod "f9be1452-095c-4dfc-9862-49ef80ac8349" (UID: "f9be1452-095c-4dfc-9862-49ef80ac8349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:26.615784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.615747    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh" pod "f9be1452-095c-4dfc-9862-49ef80ac8349" (UID: "f9be1452-095c-4dfc-9862-49ef80ac8349"). InnerVolumeSpecName "kube-api-access-s9bqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:26.716430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.716388    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9be1452-095c-4dfc-9862-49ef80ac8349-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:26.716430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.716428    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9bqh\" (UniqueName: \"kubernetes.io/projected/f9be1452-095c-4dfc-9862-49ef80ac8349-kube-api-access-s9bqh\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:26.769569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:26.769464    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:27.267359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.267329    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"
Nov 04 01:16:27.267845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.267358    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml" event={"ID":"b127ba27-cb9c-472b-ba77-c227b5962076","Type":"ContainerDied","Data":"3d751d5db67613631fc2f0e4d0f653a7aceef5915d5ca4a4758afd02a9cf2716"}
Nov 04 01:16:27.267845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.267403    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d751d5db67613631fc2f0e4d0f653a7aceef5915d5ca4a4758afd02a9cf2716"
Nov 04 01:16:27.268961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.268939    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"
Nov 04 01:16:27.269109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.268959    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2" event={"ID":"f9be1452-095c-4dfc-9862-49ef80ac8349","Type":"ContainerDied","Data":"408228046ecc440c5f8b266940328ab1cb3420349fb063013bd2de3341527c6e"}
Nov 04 01:16:27.269109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.269011    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408228046ecc440c5f8b266940328ab1cb3420349fb063013bd2de3341527c6e"
Nov 04 01:16:27.522817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.521929    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:27.720059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:27.720022    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:28.424608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:28.424444    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:29.456131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:29.446944    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:29.950014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:29.943459    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:31.229014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.228308    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:31.277547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.277508    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:31.277912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.277879    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:31.326787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.326734    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:31.474432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.474357    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:16:31.632145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.632090    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:31.633073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.633031    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:31.835582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.835534    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:31.948196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:31.948098    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:32.032156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.032097    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:32.091130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.060176    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:16:32.253059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.231653    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:16:32.253059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.233202    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:16:32.314023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.288306    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:32.756178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.340856    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:16:32.761018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.372132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:32.761335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.417726    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:32.766577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.626160    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:16:32.772308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.673697    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:32.795199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.743456    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:32.999074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.981133    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.009413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:32.988356    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:33.101128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.101075    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.164017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.115699    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.247131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.247084    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.335216    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.335364    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.335388    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"]
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.336321    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f9be1452-095c-4dfc-9862-49ef80ac8349" containerName="revoke-keys"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.336341    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9be1452-095c-4dfc-9862-49ef80ac8349" containerName="revoke-keys"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.336375    5797 cpu_manager.go:414] "containerMap: removing container" podUID="b127ba27-cb9c-472b-ba77-c227b5962076" containerName="revoke-keys"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.336382    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b127ba27-cb9c-472b-ba77-c227b5962076" containerName="revoke-keys"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.336560    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9be1452-095c-4dfc-9862-49ef80ac8349" containerName="revoke-keys"
Nov 04 01:16:33.377072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.336581    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="b127ba27-cb9c-472b-ba77-c227b5962076" containerName="revoke-keys"
Nov 04 01:16:33.505363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.505329    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:16:33.507170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.507145    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:16:33.533694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.533668    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:16:33.598168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.598138    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:16:33.598469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.598449    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:16:33.607018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.572389    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.739286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.608613    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"]
Nov 04 01:16:33.803274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.803213    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.859014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.803532    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.860244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.803562    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.860671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.808791    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.876207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.808821    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:16:33.876207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.845790    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:33.876207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.859393    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"]
Nov 04 01:16:33.876207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.859485    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:33.877941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.877908    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.878883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.878858    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.897369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.897334    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.897653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.897633    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.897843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.897826    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwbv\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.897966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.897952    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.898247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.898226    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.898368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.898354    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:33.898473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:33.898460    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.003549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.003448    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:34.198848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.185979    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwbv\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.198848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.186054    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.198848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.186165    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.198848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.186207    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.200138    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.200508    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"]
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.200609    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202143    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202228    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202282    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202314    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202380    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202454    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202505    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202587    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202716    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202759    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202840    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fg4\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.202936    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.203044    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.203123    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.282430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.279551    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.284889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.284644    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.305407 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.305377    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"]
Nov 04 01:16:34.466131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.449305    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:16:34.616789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.450909    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.617081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.490518    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:16:34.638067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.491037    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.694341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.491464    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.694589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.646307    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.694764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.694741    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.694923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.694904    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.695467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.695442    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.695646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.695627    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.697148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.697106    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.697408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.697385    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.714019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.697569    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fg4\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.715336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.715261    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.724488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.715901    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.783699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.646960    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.803405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.803369    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.988239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.805405    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.988239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.805835    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.988239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.822117    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:34.988239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.979925    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.988239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.981681    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:34.988239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.984759    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:35.005479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:34.998606    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.008938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.007798    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:35.076015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.075162    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:16:35.103204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.103145    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:35.115197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.115163    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwbv\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:35.132396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.132361    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:35.224191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.224153    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:16:35.225779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.225747    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:35.284969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.284879    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:16:35.285180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.285003    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:16:35.285180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.285033    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"]
Nov 04 01:16:35.285180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.285137    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:16:35.285180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.285173    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:16:35.285400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.285215    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:16:35.289866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.289838    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:16:35.303006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.301745    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fg4\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:35.321221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.321191    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:16:35.331402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331238    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllhc\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331320    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331417    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331446    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331518    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331598    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331646    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331774    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.331856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.331834    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.343717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.343676    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:35.395612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.395555    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433317    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllhc\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433440    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433513    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433552    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433651    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433751    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.433810    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.434080    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.435608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.434174    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.439461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.437341    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.439461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.438131    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.439461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.438393    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.447731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.447705    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.447848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.447782    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.470822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.466631    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllhc\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.470822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.467056    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.470822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.470760    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.472820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.472795    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy\") pod \"e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") " pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.510458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.510425    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:35.568452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.568373    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:35.690713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.690682    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"]
Nov 04 01:16:35.691122 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:35.691092    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d17dd83_848f_480e_b510_3580fab17a1a.slice/crio-27bdbb216f2b7d6babebf0e5ccea855cff314d0fbd7a385c2d6a2265fdacd681 WatchSource:0}: Error finding container 27bdbb216f2b7d6babebf0e5ccea855cff314d0fbd7a385c2d6a2265fdacd681: Status 404 returned error can't find the container with id 27bdbb216f2b7d6babebf0e5ccea855cff314d0fbd7a385c2d6a2265fdacd681
Nov 04 01:16:35.695506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.695468    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.695734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.695711    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.695935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.695913    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.754657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.754604    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"]
Nov 04 01:16:35.761725 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:35.761681    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262bf953_f280_410a_b214_67b2af0cd83c.slice/crio-100369bdeaef534ab7bb8fe01824b96fc63f494c7492851a06972acf32c7b68a WatchSource:0}: Error finding container 100369bdeaef534ab7bb8fe01824b96fc63f494c7492851a06972acf32c7b68a: Status 404 returned error can't find the container with id 100369bdeaef534ab7bb8fe01824b96fc63f494c7492851a06972acf32c7b68a
Nov 04 01:16:35.764959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.764919    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.765166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.765052    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.765166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.765125    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.927814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.923340    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"]
Nov 04 01:16:35.937016 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:35.929125    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495c9d92_defe_4156_9c45_8e6ed42e7b5b.slice/crio-33670b22d57e05511f97c23a90e42c2f53247368377d9a712b3788fe1f38f95d WatchSource:0}: Error finding container 33670b22d57e05511f97c23a90e42c2f53247368377d9a712b3788fe1f38f95d: Status 404 returned error can't find the container with id 33670b22d57e05511f97c23a90e42c2f53247368377d9a712b3788fe1f38f95d
Nov 04 01:16:35.955941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.955058    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.955941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.955396    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:35.955941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:35.955650    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:16:36.347455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.347415    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" event={"ID":"262bf953-f280-410a-b214-67b2af0cd83c","Type":"ContainerStarted","Data":"afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719"}
Nov 04 01:16:36.347781 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.347753    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" event={"ID":"262bf953-f280-410a-b214-67b2af0cd83c","Type":"ContainerStarted","Data":"100369bdeaef534ab7bb8fe01824b96fc63f494c7492851a06972acf32c7b68a"}
Nov 04 01:16:36.351962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.351935    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" event={"ID":"495c9d92-defe-4156-9c45-8e6ed42e7b5b","Type":"ContainerStarted","Data":"16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741"}
Nov 04 01:16:36.353125 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.353090    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" event={"ID":"495c9d92-defe-4156-9c45-8e6ed42e7b5b","Type":"ContainerStarted","Data":"33670b22d57e05511f97c23a90e42c2f53247368377d9a712b3788fe1f38f95d"}
Nov 04 01:16:36.356700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.356673    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" event={"ID":"5d17dd83-848f-480e-b510-3580fab17a1a","Type":"ContainerStarted","Data":"52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88"}
Nov 04 01:16:36.356860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.356842    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" event={"ID":"5d17dd83-848f-480e-b510-3580fab17a1a","Type":"ContainerStarted","Data":"27bdbb216f2b7d6babebf0e5ccea855cff314d0fbd7a385c2d6a2265fdacd681"}
Nov 04 01:16:36.379679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.379561    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podStartSLOduration=4.37954299 podStartE2EDuration="4.37954299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:36.376904832 +0000 UTC m=+784.018811097" watchObservedRunningTime="2026-11-04 01:16:36.37954299 +0000 UTC m=+784.021449233"
Nov 04 01:16:36.409220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.407688    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podStartSLOduration=4.407668503 podStartE2EDuration="4.407668503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:36.404889101 +0000 UTC m=+784.046795385" watchObservedRunningTime="2026-11-04 01:16:36.407668503 +0000 UTC m=+784.049574755"
Nov 04 01:16:36.409405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.409219    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:36.409607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.409585    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:36.409702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.409630    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:36.433856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.433802    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podStartSLOduration=4.433786605 podStartE2EDuration="4.433786605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:36.430371546 +0000 UTC m=+784.072277817" watchObservedRunningTime="2026-11-04 01:16:36.433786605 +0000 UTC m=+784.075692854"
Nov 04 01:16:36.569210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.569174    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:36.772446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:36.772354    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:37.345409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.344890    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:37.345558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.345457    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:37.345558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.345498    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:37.397014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.396134    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:37.397014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.396194    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:37.599081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.571727    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 04 01:16:37.599709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.599666    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 04 01:16:37.944014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:37.943032    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:38.344269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.344185    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:38.344269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.344246    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:38.418012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.409754    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:38.418012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.409825    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:38.569447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.569406    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 04 01:16:38.569732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.569696    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 04 01:16:38.726018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.721110    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"]
Nov 04 01:16:38.745694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.744535    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:38.787016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.748743    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"]
Nov 04 01:16:38.846037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.787827    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:16:38.846037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.788574    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:16:38.846037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.844038    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5xz\" (UniqueName: \"kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:38.846037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.845073    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:38.846037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.845127    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:38.956013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.952202    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:38.956013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.952307    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:38.956013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:38.952577    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5xz\" (UniqueName: \"kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:39.102011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.076605    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:39.131026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.128453    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5xz\" (UniqueName: \"kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:39.176117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.176079    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:39.194592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.194556    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037160aa-85fzw\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:39.357158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.354586    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:39.357158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.354651    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:39.401412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.401377    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:39.402389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.402353    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:39.437482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.436234    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:39.572721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.572684    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 04 01:16:39.573013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:39.572955    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 04 01:16:40.333013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.330485    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"]
Nov 04 01:16:40.358014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.352368    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:40.358014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.352428    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:40.397131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.397093    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:40.397386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.397355    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:40.587013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.577010    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 04 01:16:40.587013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.577069    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 04 01:16:40.610016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.597674    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw" event={"ID":"134d6eab-526f-4cbe-b939-93ac1104337e","Type":"ContainerStarted","Data":"92344fb8712b98a8e2285a04137e82dac5d5bf1c0527ef9f499be1b59e5fbb45"}
Nov 04 01:16:40.610902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:40.610821    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:41.344581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.344540    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:41.345099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.345065    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:41.414015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.409674    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:41.414015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.409735    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:41.573028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.572855    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 04 01:16:41.573028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.572927    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 04 01:16:41.635949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.635913    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw" event={"ID":"134d6eab-526f-4cbe-b939-93ac1104337e","Type":"ContainerStarted","Data":"d2aac2ad7d9a42082588c3f1f3034f90ed159eca313c8e987981a0fb4ad7648f"}
Nov 04 01:16:41.819014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:41.818799    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:42.344361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.344329    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:42.344515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.344378    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:42.398699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.398265    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:42.398699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.398323    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:42.569488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.569446    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 04 01:16:42.569798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.569767    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 04 01:16:42.650126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.649861    5797 generic.go:356] "Generic (PLEG): container finished" podID="134d6eab-526f-4cbe-b939-93ac1104337e" containerID="d2aac2ad7d9a42082588c3f1f3034f90ed159eca313c8e987981a0fb4ad7648f" exitCode=0
Nov 04 01:16:42.650126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.649960    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw" event={"ID":"134d6eab-526f-4cbe-b939-93ac1104337e","Type":"ContainerDied","Data":"d2aac2ad7d9a42082588c3f1f3034f90ed159eca313c8e987981a0fb4ad7648f"}
Nov 04 01:16:42.861954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.861637    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:42.925475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.925441    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:42.973264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.973220    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca\") pod \"134d6eab-526f-4cbe-b939-93ac1104337e\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") "
Nov 04 01:16:42.973922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.973900    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz\" (UniqueName: \"kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz\") pod \"134d6eab-526f-4cbe-b939-93ac1104337e\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") "
Nov 04 01:16:42.974478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.974456    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca\") pod \"134d6eab-526f-4cbe-b939-93ac1104337e\" (UID: \"134d6eab-526f-4cbe-b939-93ac1104337e\") "
Nov 04 01:16:42.975952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:42.973841    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca" pod "134d6eab-526f-4cbe-b939-93ac1104337e" (UID: "134d6eab-526f-4cbe-b939-93ac1104337e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:43.017292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.017251    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz" pod "134d6eab-526f-4cbe-b939-93ac1104337e" (UID: "134d6eab-526f-4cbe-b939-93ac1104337e"). InnerVolumeSpecName "kube-api-access-7x5xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:43.019719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.019448    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca" pod "134d6eab-526f-4cbe-b939-93ac1104337e" (UID: "134d6eab-526f-4cbe-b939-93ac1104337e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:43.078103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.077815    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:43.078103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.077855    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5xz\" (UniqueName: \"kubernetes.io/projected/134d6eab-526f-4cbe-b939-93ac1104337e-kube-api-access-7x5xz\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:43.078103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.077874    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134d6eab-526f-4cbe-b939-93ac1104337e-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:43.345099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.345009    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 04 01:16:43.345099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.345075    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 04 01:16:43.396307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.396272    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 04 01:16:43.396503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.396323    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 04 01:16:43.667739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.667692    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw" event={"ID":"134d6eab-526f-4cbe-b939-93ac1104337e","Type":"ContainerDied","Data":"92344fb8712b98a8e2285a04137e82dac5d5bf1c0527ef9f499be1b59e5fbb45"}
Nov 04 01:16:43.668435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.667760    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92344fb8712b98a8e2285a04137e82dac5d5bf1c0527ef9f499be1b59e5fbb45"
Nov 04 01:16:43.668435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.667840    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"
Nov 04 01:16:43.898027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:43.897563    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:44.019192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.016704    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:44.019192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.018527    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:44.019968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.019951    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:16:44.434752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.434718    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"]
Nov 04 01:16:44.436728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.436687    5797 cpu_manager.go:414] "containerMap: removing container" podUID="134d6eab-526f-4cbe-b939-93ac1104337e" containerName="revoke-keys"
Nov 04 01:16:44.436728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.436711    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134d6eab-526f-4cbe-b939-93ac1104337e" containerName="revoke-keys"
Nov 04 01:16:44.436912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.436818    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="134d6eab-526f-4cbe-b939-93ac1104337e" containerName="revoke-keys"
Nov 04 01:16:44.446851 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.446758    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"]
Nov 04 01:16:44.447009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.446878    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.462093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.451218    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:16:44.462093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.451565    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:16:44.506377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.506314    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:44.508012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.507972    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:44.510092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.510070    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:16:44.613155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.613048    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:44.614131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.614109    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:44.615208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.615186    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:16:44.626223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.625655    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ztf\" (UniqueName: \"kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.626223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.625754    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.626223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.625911    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.727654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.727553    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.728192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.727740    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.728192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.728146    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ztf\" (UniqueName: \"kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.728353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.728332    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.730487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.730461    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.740626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.740592    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ztf\" (UniqueName: \"kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf\") pod \"maas-api-revoke-sub-c4a5d864-62bfl\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.791918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.791882    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:44.925727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:44.925557    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:45.112291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:45.112259    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"]
Nov 04 01:16:45.112549 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:45.112512    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef2953c_0652_404d_8386_607b0f909068.slice/crio-ab574e96b126d69daed8e6af8a177865d69495fa2d12d7b83f4f37ac6de80bdb WatchSource:0}: Error finding container ab574e96b126d69daed8e6af8a177865d69495fa2d12d7b83f4f37ac6de80bdb: Status 404 returned error can't find the container with id ab574e96b126d69daed8e6af8a177865d69495fa2d12d7b83f4f37ac6de80bdb
Nov 04 01:16:45.703591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:45.703556    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl" event={"ID":"2ef2953c-0652-404d-8386-607b0f909068","Type":"ContainerStarted","Data":"21867df1877375196ba601c1f58d177e0bfc9ac5bc2ce33f016d972fbd51f6c4"}
Nov 04 01:16:45.703743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:45.703653    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl" event={"ID":"2ef2953c-0652-404d-8386-607b0f909068","Type":"ContainerStarted","Data":"ab574e96b126d69daed8e6af8a177865d69495fa2d12d7b83f4f37ac6de80bdb"}
Nov 04 01:16:45.730079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:45.730022    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl" podStartSLOduration=1.730004283 podStartE2EDuration="1.730004283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:45.723343237 +0000 UTC m=+793.365249488" watchObservedRunningTime="2026-11-04 01:16:45.730004283 +0000 UTC m=+793.371910535"
Nov 04 01:16:45.811127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:45.811093    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:46.568527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.568431    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:46.713213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.713181    5797 generic.go:356] "Generic (PLEG): container finished" podID="2ef2953c-0652-404d-8386-607b0f909068" containerID="21867df1877375196ba601c1f58d177e0bfc9ac5bc2ce33f016d972fbd51f6c4" exitCode=0
Nov 04 01:16:46.713363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.713262    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl" event={"ID":"2ef2953c-0652-404d-8386-607b0f909068","Type":"ContainerDied","Data":"21867df1877375196ba601c1f58d177e0bfc9ac5bc2ce33f016d972fbd51f6c4"}
Nov 04 01:16:46.885797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.885764    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"]
Nov 04 01:16:46.890230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.890206    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:46.897881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.897853    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"]
Nov 04 01:16:46.976157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.976125    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:46.976319 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.976266    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlv4\" (UniqueName: \"kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:46.976387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:46.976372    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.077438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.077395    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlv4\" (UniqueName: \"kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.077610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.077537    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.077746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.077711    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.078347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.078324    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.080030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.080005    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.091315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.091288    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlv4\" (UniqueName: \"kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4\") pod \"maas-api-revoke-sub-bcbcdaea-6plvs\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.112571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.112540    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:47.194362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.194285    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"]
Nov 04 01:16:47.200788 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.200201    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.209086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.208384    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:47.215941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.215866    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"]
Nov 04 01:16:47.280353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.280316    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbr7l\" (UniqueName: \"kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.280574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.280554    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.280654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.280637    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.315145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.315047    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:47.382920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.382879    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbr7l\" (UniqueName: \"kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.383173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.383152    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.383748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.383724    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.384705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.384538    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.385880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.385860    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.438128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.424737    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbr7l\" (UniqueName: \"kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l\") pod \"maas-api-revoke-sub-dbdae9ea-jkxkz\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.513966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.513866    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:47.521397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.521370    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"]
Nov 04 01:16:47.524612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.524586    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:47.524822 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:47.524800    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf05aeff_5a1b_4ee9_ae25_ddc8a30815b7.slice/crio-3919ab297f74f126d41951389a20aef59720de1dbe038b1f85d578c2338b21e7 WatchSource:0}: Error finding container 3919ab297f74f126d41951389a20aef59720de1dbe038b1f85d578c2338b21e7: Status 404 returned error can't find the container with id 3919ab297f74f126d41951389a20aef59720de1dbe038b1f85d578c2338b21e7
Nov 04 01:16:47.722545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.722508    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs" event={"ID":"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7","Type":"ContainerStarted","Data":"3919ab297f74f126d41951389a20aef59720de1dbe038b1f85d578c2338b21e7"}
Nov 04 01:16:47.824034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.824004    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"]
Nov 04 01:16:47.893344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.893321    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:47.992443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.992409    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca\") pod \"2ef2953c-0652-404d-8386-607b0f909068\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") "
Nov 04 01:16:47.992616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.992451    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf\" (UniqueName: \"kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf\") pod \"2ef2953c-0652-404d-8386-607b0f909068\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") "
Nov 04 01:16:47.992688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.992671    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca\") pod \"2ef2953c-0652-404d-8386-607b0f909068\" (UID: \"2ef2953c-0652-404d-8386-607b0f909068\") "
Nov 04 01:16:47.993038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.992979    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca" pod "2ef2953c-0652-404d-8386-607b0f909068" (UID: "2ef2953c-0652-404d-8386-607b0f909068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:47.993304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:47.993288    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef2953c-0652-404d-8386-607b0f909068-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:48.047623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.047581    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca" pod "2ef2953c-0652-404d-8386-607b0f909068" (UID: "2ef2953c-0652-404d-8386-607b0f909068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:48.047857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.047834    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf" pod "2ef2953c-0652-404d-8386-607b0f909068" (UID: "2ef2953c-0652-404d-8386-607b0f909068"). InnerVolumeSpecName "kube-api-access-d4ztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:48.094543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.094513    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef2953c-0652-404d-8386-607b0f909068-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:48.094543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.094543    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4ztf\" (UniqueName: \"kubernetes.io/projected/2ef2953c-0652-404d-8386-607b0f909068-kube-api-access-d4ztf\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:48.406716 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.406677    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:48.732758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.732723    5797 generic.go:356] "Generic (PLEG): container finished" podID="cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" containerID="5a7bce3d2f9f24213d1c23b328b8759542abcf2cab9be0ed401d626481531f17" exitCode=0
Nov 04 01:16:48.732915 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.732803    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs" event={"ID":"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7","Type":"ContainerDied","Data":"5a7bce3d2f9f24213d1c23b328b8759542abcf2cab9be0ed401d626481531f17"}
Nov 04 01:16:48.734181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.734164    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"
Nov 04 01:16:48.734321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.734179    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl" event={"ID":"2ef2953c-0652-404d-8386-607b0f909068","Type":"ContainerDied","Data":"ab574e96b126d69daed8e6af8a177865d69495fa2d12d7b83f4f37ac6de80bdb"}
Nov 04 01:16:48.734321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.734235    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab574e96b126d69daed8e6af8a177865d69495fa2d12d7b83f4f37ac6de80bdb"
Nov 04 01:16:48.735951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.735930    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz" event={"ID":"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82","Type":"ContainerStarted","Data":"d01bdcaee6f20cfa8db924de1c7e5722c845c6f9a5f6829a83ce5d9fe03a352b"}
Nov 04 01:16:48.736060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.735976    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz" event={"ID":"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82","Type":"ContainerStarted","Data":"a31d01717f86de998a8f1a5451e7c19405c49299fad720c3d48d41f09052bdf8"}
Nov 04 01:16:48.783917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.783868    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz" podStartSLOduration=1.783854543 podStartE2EDuration="1.783854543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:48.778451976 +0000 UTC m=+796.420358246" watchObservedRunningTime="2026-11-04 01:16:48.783854543 +0000 UTC m=+796.425760792"
Nov 04 01:16:48.989074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.988999    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"]
Nov 04 01:16:48.989675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.989661    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2ef2953c-0652-404d-8386-607b0f909068" containerName="revoke-keys"
Nov 04 01:16:48.989721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.989677    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef2953c-0652-404d-8386-607b0f909068" containerName="revoke-keys"
Nov 04 01:16:48.989813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.989804    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef2953c-0652-404d-8386-607b0f909068" containerName="revoke-keys"
Nov 04 01:16:48.993584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:48.993566    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.001535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.001508    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"]
Nov 04 01:16:49.109250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.109217    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.109439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.109414    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.109607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.109595    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjsj\" (UniqueName: \"kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.211641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.211606    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.211863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.211843    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjsj\" (UniqueName: \"kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.211908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.211888    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.212330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.212312    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.214416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.214392    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.225535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.225508    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjsj\" (UniqueName: \"kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj\") pod \"maas-api-revoke-sub-7b8d5271-7p2h2\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.321071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.320276    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:49.375241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.375193    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:49.660460 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:49.660396    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7b644c_fbc0_436d_8fd1_88ea9b9c3a0f.slice/crio-a18555ef6b18fd47d2779db9720961ede11491127eb7fe6119a0c23b4ae570d2 WatchSource:0}: Error finding container a18555ef6b18fd47d2779db9720961ede11491127eb7fe6119a0c23b4ae570d2: Status 404 returned error can't find the container with id a18555ef6b18fd47d2779db9720961ede11491127eb7fe6119a0c23b4ae570d2
Nov 04 01:16:49.660749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.660726    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"]
Nov 04 01:16:49.745273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.745235    5797 generic.go:356] "Generic (PLEG): container finished" podID="d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" containerID="d01bdcaee6f20cfa8db924de1c7e5722c845c6f9a5f6829a83ce5d9fe03a352b" exitCode=0
Nov 04 01:16:49.745449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.745325    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz" event={"ID":"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82","Type":"ContainerDied","Data":"d01bdcaee6f20cfa8db924de1c7e5722c845c6f9a5f6829a83ce5d9fe03a352b"}
Nov 04 01:16:49.746706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.746676    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2" event={"ID":"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f","Type":"ContainerStarted","Data":"a18555ef6b18fd47d2779db9720961ede11491127eb7fe6119a0c23b4ae570d2"}
Nov 04 01:16:49.913953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:49.913891    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:50.042880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.042845    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca\") pod \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") "
Nov 04 01:16:50.043390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.042894    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4\" (UniqueName: \"kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4\") pod \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") "
Nov 04 01:16:50.043390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.043168    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca\") pod \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\" (UID: \"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7\") "
Nov 04 01:16:50.043390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.043317    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca" pod "cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" (UID: "cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:50.043793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.043774    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:50.045078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.045052    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca" pod "cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" (UID: "cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:50.045234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.045212    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4" pod "cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" (UID: "cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7"). InnerVolumeSpecName "kube-api-access-fhlv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:50.145142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.145109    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:50.145142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.145143    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlv4\" (UniqueName: \"kubernetes.io/projected/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7-kube-api-access-fhlv4\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:50.267729 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.267685    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:50.755635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.755606    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"
Nov 04 01:16:50.755799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.755634    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs" event={"ID":"cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7","Type":"ContainerDied","Data":"3919ab297f74f126d41951389a20aef59720de1dbe038b1f85d578c2338b21e7"}
Nov 04 01:16:50.755799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.755691    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3919ab297f74f126d41951389a20aef59720de1dbe038b1f85d578c2338b21e7"
Nov 04 01:16:50.757312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.757283    5797 generic.go:356] "Generic (PLEG): container finished" podID="1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" containerID="033c923ca7e1471ea227770dc2815e7ca0908b4c625eb43c6b9959826bcdeeec" exitCode=0
Nov 04 01:16:50.757430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.757312    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2" event={"ID":"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f","Type":"ContainerDied","Data":"033c923ca7e1471ea227770dc2815e7ca0908b4c625eb43c6b9959826bcdeeec"}
Nov 04 01:16:50.937924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:50.937898    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:51.058774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.058742    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca\") pod \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") "
Nov 04 01:16:51.059278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.058894    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l\" (UniqueName: \"kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l\") pod \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") "
Nov 04 01:16:51.059278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.059041    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca\") pod \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\" (UID: \"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82\") "
Nov 04 01:16:51.059278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.059092    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca" pod "d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" (UID: "d4e8cf0c-cdbf-40ea-ba23-05486ceedf82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:51.059550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.059532    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:51.061069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.061037    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca" pod "d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" (UID: "d4e8cf0c-cdbf-40ea-ba23-05486ceedf82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:51.061209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.061185    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l" pod "d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" (UID: "d4e8cf0c-cdbf-40ea-ba23-05486ceedf82"). InnerVolumeSpecName "kube-api-access-mbr7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:51.167065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.161707    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbr7l\" (UniqueName: \"kubernetes.io/projected/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-kube-api-access-mbr7l\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:51.167065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.161749    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:51.170603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.170573    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:51.766929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.766897    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"
Nov 04 01:16:51.767154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.766925    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz" event={"ID":"d4e8cf0c-cdbf-40ea-ba23-05486ceedf82","Type":"ContainerDied","Data":"a31d01717f86de998a8f1a5451e7c19405c49299fad720c3d48d41f09052bdf8"}
Nov 04 01:16:51.767154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.766979    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31d01717f86de998a8f1a5451e7c19405c49299fad720c3d48d41f09052bdf8"
Nov 04 01:16:51.949381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.949284    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:51.997063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:51.997035    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:52.075900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.075807    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca\") pod \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") "
Nov 04 01:16:52.076416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.076000    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj\" (UniqueName: \"kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj\") pod \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") "
Nov 04 01:16:52.076416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.076122    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca\") pod \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\" (UID: \"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f\") "
Nov 04 01:16:52.076537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.076499    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca" pod "1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" (UID: "1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:52.077022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.076959    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:52.078198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.078163    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca" pod "1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" (UID: "1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:52.078301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.078272    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj" pod "1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" (UID: "1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f"). InnerVolumeSpecName "kube-api-access-qjjsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:52.178480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.178450    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjsj\" (UniqueName: \"kubernetes.io/projected/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-kube-api-access-qjjsj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:52.178480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.178477    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:52.724708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.724644    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:52.777514 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.777438    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"
Nov 04 01:16:52.777514 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.777462    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2" event={"ID":"1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f","Type":"ContainerDied","Data":"a18555ef6b18fd47d2779db9720961ede11491127eb7fe6119a0c23b4ae570d2"}
Nov 04 01:16:52.777719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:52.777519    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18555ef6b18fd47d2779db9720961ede11491127eb7fe6119a0c23b4ae570d2"
Nov 04 01:16:53.489657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.489536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:53.705464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.705435    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"]
Nov 04 01:16:53.706126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706109    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" containerName="revoke-keys"
Nov 04 01:16:53.706126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706126    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" containerName="revoke-keys"
Nov 04 01:16:53.706296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706153    5797 cpu_manager.go:414] "containerMap: removing container" podUID="d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" containerName="revoke-keys"
Nov 04 01:16:53.706296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706159    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" containerName="revoke-keys"
Nov 04 01:16:53.706296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706173    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" containerName="revoke-keys"
Nov 04 01:16:53.706296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706178    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" containerName="revoke-keys"
Nov 04 01:16:53.706296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706283    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" containerName="revoke-keys"
Nov 04 01:16:53.706296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706297    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" containerName="revoke-keys"
Nov 04 01:16:53.706475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.706304    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" containerName="revoke-keys"
Nov 04 01:16:53.710693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.710673    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.719672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.719107    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:16:53.719672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.719211    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:16:53.723081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.723058    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"]
Nov 04 01:16:53.815465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.815382    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.815465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.815438    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.815641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.815542    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8bz\" (UniqueName: \"kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.916694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.916662    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.916896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.916733    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.916896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.916794    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8bz\" (UniqueName: \"kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.917336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.917311    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.919216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.919188    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:53.931305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:53.931274    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8bz\" (UniqueName: \"kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz\") pod \"maas-api-revoke-sub-e5c865c0-xj9gm\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:54.028934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.028901    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:54.191237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.191200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:54.313322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.313295    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"]
Nov 04 01:16:54.313957 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:16:54.313935    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e1bcf3_7f5e_48bd_8978_558783c36c4a.slice/crio-490bbd1e2744e61787f3d36db54b245662fde4d79c1e3a41fa4bea29deba16c2 WatchSource:0}: Error finding container 490bbd1e2744e61787f3d36db54b245662fde4d79c1e3a41fa4bea29deba16c2: Status 404 returned error can't find the container with id 490bbd1e2744e61787f3d36db54b245662fde4d79c1e3a41fa4bea29deba16c2
Nov 04 01:16:54.807827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.807793    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm" event={"ID":"72e1bcf3-7f5e-48bd-8978-558783c36c4a","Type":"ContainerStarted","Data":"9244877619ef53e0766dcb5da0759296fd4f96828fadc44fbd741f1c16e049f5"}
Nov 04 01:16:54.809638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.809607    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm" event={"ID":"72e1bcf3-7f5e-48bd-8978-558783c36c4a","Type":"ContainerStarted","Data":"490bbd1e2744e61787f3d36db54b245662fde4d79c1e3a41fa4bea29deba16c2"}
Nov 04 01:16:54.839815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.839747    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm" podStartSLOduration=1.8397263700000002 podStartE2EDuration="1.83972637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:54.837432717 +0000 UTC m=+802.479338969" watchObservedRunningTime="2026-11-04 01:16:54.83972637 +0000 UTC m=+802.481632621"
Nov 04 01:16:54.846604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:54.846562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:55.558059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:55.557929    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:55.817086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:55.816958    5797 generic.go:356] "Generic (PLEG): container finished" podID="72e1bcf3-7f5e-48bd-8978-558783c36c4a" containerID="9244877619ef53e0766dcb5da0759296fd4f96828fadc44fbd741f1c16e049f5" exitCode=0
Nov 04 01:16:55.817086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:55.817041    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm" event={"ID":"72e1bcf3-7f5e-48bd-8978-558783c36c4a","Type":"ContainerDied","Data":"9244877619ef53e0766dcb5da0759296fd4f96828fadc44fbd741f1c16e049f5"}
Nov 04 01:16:56.253737 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:16:56.253702    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:16:56.356905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:56.356869    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:57.008958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.008931    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:57.038054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.035302    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:57.174107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.174067    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca\") pod \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") "
Nov 04 01:16:57.174311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.174139    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz\" (UniqueName: \"kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz\") pod \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") "
Nov 04 01:16:57.174311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.174295    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca\") pod \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\" (UID: \"72e1bcf3-7f5e-48bd-8978-558783c36c4a\") "
Nov 04 01:16:57.174482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.174453    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca" pod "72e1bcf3-7f5e-48bd-8978-558783c36c4a" (UID: "72e1bcf3-7f5e-48bd-8978-558783c36c4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:16:57.174881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.174859    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:57.176371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.176343    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz" pod "72e1bcf3-7f5e-48bd-8978-558783c36c4a" (UID: "72e1bcf3-7f5e-48bd-8978-558783c36c4a"). InnerVolumeSpecName "kube-api-access-8d8bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:16:57.176452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.176348    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca" pod "72e1bcf3-7f5e-48bd-8978-558783c36c4a" (UID: "72e1bcf3-7f5e-48bd-8978-558783c36c4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:16:57.276467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.276434    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8bz\" (UniqueName: \"kubernetes.io/projected/72e1bcf3-7f5e-48bd-8978-558783c36c4a-kube-api-access-8d8bz\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:57.276467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.276466    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e1bcf3-7f5e-48bd-8978-558783c36c4a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:16:57.786307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.786278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:57.876111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.868544    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm" event={"ID":"72e1bcf3-7f5e-48bd-8978-558783c36c4a","Type":"ContainerDied","Data":"490bbd1e2744e61787f3d36db54b245662fde4d79c1e3a41fa4bea29deba16c2"}
Nov 04 01:16:57.876111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.868627    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490bbd1e2744e61787f3d36db54b245662fde4d79c1e3a41fa4bea29deba16c2"
Nov 04 01:16:57.876111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:57.868716    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"
Nov 04 01:16:58.480026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:58.479982    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:16:59.364681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:16:59.364134    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:00.338402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:00.338370    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:01.195010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:01.190366    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:01.979615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:01.979574    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"]
Nov 04 01:17:01.988021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:01.981669    5797 cpu_manager.go:414] "containerMap: removing container" podUID="72e1bcf3-7f5e-48bd-8978-558783c36c4a" containerName="revoke-keys"
Nov 04 01:17:01.988276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:01.988250    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e1bcf3-7f5e-48bd-8978-558783c36c4a" containerName="revoke-keys"
Nov 04 01:17:01.988627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:01.988607    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e1bcf3-7f5e-48bd-8978-558783c36c4a" containerName="revoke-keys"
Nov 04 01:17:02.052023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.019241    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.052023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.021377    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:02.160029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.159192    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:02.160029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.159615    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:02.189523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.189093    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:02.221891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.221842    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278071    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278249    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278285    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278324    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmd5\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278375    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278615    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278696    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.282552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.278721    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.344122 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.302101    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:02.344122 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.326253    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"]
Nov 04 01:17:02.355757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.355153    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 04 01:17:02.411598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.411558    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.412055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.412029    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.412301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.412284    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.412680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.412083    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.412680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.412640    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.413134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.413113    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.422299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.422168    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.447214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.413519    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.447214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.442255    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.449252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.449199    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.450262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.449771    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.516335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.450505    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmd5\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.517768 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.517732    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.522592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.518966    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.522766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.520764    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.601040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.599435    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.784271 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.780855    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmd5\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.784271 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.781484    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo\") pod \"e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") " pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.794938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.794846    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"]
Nov 04 01:17:02.870021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.866397    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:02.870021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.867847    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:02.881756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.874069    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:17:02.881756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.874317    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:02.899651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.899399    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:02.993239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.993200    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"]
Nov 04 01:17:03.046165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:02.999192    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.093619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.093523    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 04 01:17:03.144236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.144193    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.144595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.144562    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.148222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.148146    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.158014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.152210    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.180790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.180750    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.181215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.162734    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:03.181480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.162691    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:03.204347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.204311    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:03.224365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.206376    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:17:03.224365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.206470    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:17:03.225176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.225138    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:17:03.225447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.225427    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.225649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.225607    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsd6p\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.225971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.225952    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.226210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.226191    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.419553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.419508    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.419876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.419849    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsd6p\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.420155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.420135    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.420305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.420285    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.420599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.420581    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.420750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.420734    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.420898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.420880    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.421183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.421160    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.421332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.421316    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.443022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.421941    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.471511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.444090    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.472425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.449078    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:03.472882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.450630    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.473047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.451559    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.473150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.452132    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:03.473608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.473588    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:17:03.860077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.859335    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 04 01:17:03.894013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:03.892132    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.065694    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsd6p\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.143759    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.144531    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.171440    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.172387    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.174690    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.174774    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.174834    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.251783    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.281599    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:04.299161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.292423    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:04.594069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.561488    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo\") pod \"e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") " pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:04.812105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.812054    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:17:04.869123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:04.866567    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:05.038014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.035526    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.081866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.081818    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:50052\" within 1s: context deadline exceeded"
Nov 04 01:17:05.110014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.084610    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded"
Nov 04 01:17:05.136074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.111357    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:50052\" within 1s: context deadline exceeded"
Nov 04 01:17:05.269924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.159223    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.276083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.159326    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:05.276276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.159545    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded"
Nov 04 01:17:05.276390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.173414    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.276496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.173468    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.301385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.255755    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.301958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.255873    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.46:50052\" within 1s: context deadline exceeded"
Nov 04 01:17:05.302119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.297103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:05.550073 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:05.537674    5797 logging.go:55] [core] [Channel #553 SubChannel #554]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.46:50052", ServerName: "10.134.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:53896->10.134.0.46:50052: use of closed network connection"
Nov 04 01:17:05.550073 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:05.539137    5797 logging.go:55] [core] [Channel #548 SubChannel #555]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:50052", ServerName: "10.134.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:33392->10.134.0.48:50052: use of closed network connection"
Nov 04 01:17:05.550073 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:05.539238    5797 logging.go:55] [core] [Channel #547 SubChannel #552]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:50052", ServerName: "10.134.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:35186->10.134.0.51:50052: use of closed network connection"
Nov 04 01:17:05.550073 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:05.540385    5797 logging.go:55] [core] [Channel #545 SubChannel #550]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:38848->10.134.0.49:50052: use of closed network connection"
Nov 04 01:17:05.578015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.574623    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:05.581533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.578436    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.581533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.579816    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.660016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.651277    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:05.660016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.651342    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:05.660016 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:05.651493    5797 logging.go:55] [core] [Channel #549 SubChannel #551]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:39492->10.134.0.50:50052: use of closed network connection"
Nov 04 01:17:05.979081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:05.925284    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:17:06.157067 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:06.106440    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.566s"
Nov 04 01:17:06.173751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.173730    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:17:06.173927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.173915    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:17:06.178156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.178129    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:17:06.397011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.384033    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" event={"ID":"94f234fa-0b09-4589-a6ab-0f57217e890a","Type":"ContainerStarted","Data":"bbe721d10f2aeab7d237c1a5f317b81d89593e3263eae367473fde15783cff30"}
Nov 04 01:17:06.491070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.472317    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:06.529063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.529015    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:06.553146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.553089    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:06.585883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.585840    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:06.591027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.589069    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:06.645014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.632892    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:06.671932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.671885    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:06.672245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.672216    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:06.719497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.719451    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:06.748060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:06.747950    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:07.053192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.053067    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:07.317888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.317791    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"]
Nov 04 01:17:07.383356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.383321    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"]
Nov 04 01:17:07.412254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.393171    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.459218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.459187    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.473022    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.474044    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.643875    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4nb\" (UniqueName: \"kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.644427    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.588471    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.475071    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.588435    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.621565    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:07.684896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.621594    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:07.739825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.739774    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:07.740453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.740417    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 04 01:17:07.766032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.749251    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.766032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.749308    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.766032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.749448    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4nb\" (UniqueName: \"kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.766032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.750769    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:07.794029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.788742    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:17:07.899041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:07.863304    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4nb\" (UniqueName: \"kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:08.017043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.002501    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c390d9-8kwqz\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:08.069223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.069103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:08.296021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.264135    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:08.296021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.264203    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:08.296021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.285686    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:08.902054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.875638    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:08.921016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.907491    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:08.998017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.978232    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:08.998017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.979855    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:08.998017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:08.987073    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:09.049015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:09.024395    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:09.110531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:09.110484    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:09.174075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:09.035137    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:09.577325 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:09.540301    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.064s"
Nov 04 01:17:09.654025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:09.649184    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"]
Nov 04 01:17:09.909646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:09.909598    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:10.046183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.027511    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:10.046183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.042970    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:10.046183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.043188    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:10.046183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.043295    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:10.154011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.129172    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:10.202060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.139037    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:10.202060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.197125    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:10.202060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.201069    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:10.230010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.146820    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:10.230010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.149106    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:10.230010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.155297    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"]
Nov 04 01:17:10.379569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.379539    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.383332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.383293    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" event={"ID":"94f234fa-0b09-4589-a6ab-0f57217e890a","Type":"ContainerStarted","Data":"9d12511a8edc6d3f7fc2dc6a8207d513b8269f051d0af858b373540f3192108d"}
Nov 04 01:17:10.433703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.433676    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:17:10.434845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.434825    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:17:10.475309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.468955    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:17:10.475309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.469269    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:17:10.475309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.469298    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"]
Nov 04 01:17:10.506939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.490655    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.506939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.490923    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.506939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.490954    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd87n\" (UniqueName: \"kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.585618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.585587    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:17:10.602179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.599587    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.602179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.599625    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd87n\" (UniqueName: \"kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.602179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.599861    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.602179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.600520    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:17:10.602179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.601850    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:17:10.603010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.602976    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:17:10.606717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.606694    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"]
Nov 04 01:17:10.611269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.611242    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.626673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.626639    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"]
Nov 04 01:17:10.626871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.626764    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.641875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.636325    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"]
Nov 04 01:17:10.641875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.639625    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podStartSLOduration=9.639607833 podStartE2EDuration="9.639607833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:10.5690982 +0000 UTC m=+818.211004477" watchObservedRunningTime="2026-11-04 01:17:10.639607833 +0000 UTC m=+818.281514084"
Nov 04 01:17:10.644030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.644004    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.653768 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.653700    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd87n\" (UniqueName: \"kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n\") pod \"maas-api-revoke-sub-49bc74ea-m5zss\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.689202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.689133    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:10.712257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.710709    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.712257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.711038    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.712257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.711206    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lph7\" (UniqueName: \"kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.812838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.812802    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.816311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.816284    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lph7\" (UniqueName: \"kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.816943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.816918    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.817380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.815221    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.824412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.824386    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.888145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.881272    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:10.892812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.892776    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"]
Nov 04 01:17:10.893843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.893747    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lph7\" (UniqueName: \"kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7\") pod \"maas-api-revoke-sub-7a072f6b-gfbzl\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:10.904558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.902066    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:10.904558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.903799    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:10.904558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:10.903848    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:11.087015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.085614    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:11.596232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.596150    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"]
Nov 04 01:17:11.612394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.612351    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" event={"ID":"e43f7199-d45f-4e71-b5dc-7bd24a975284","Type":"ContainerStarted","Data":"e13db9797f0c4562db17be0664c1ff7ff345e38a4b91ab10c42f12c40c68f2ea"}
Nov 04 01:17:11.612526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.612442    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" event={"ID":"e43f7199-d45f-4e71-b5dc-7bd24a975284","Type":"ContainerStarted","Data":"f70edd4a820b0f7052d252b6d91c622f5e8d545218991f9409ea4207a43aa5b0"}
Nov 04 01:17:11.620227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.619968    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz" event={"ID":"9c1a3431-f7b6-4455-8877-07ce2a668170","Type":"ContainerStarted","Data":"7a8f70330c1a2f7b5baf8b5e7c63df272744b3617b49c42cd6419a616b3749c2"}
Nov 04 01:17:11.699016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.648978    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podStartSLOduration=9.648956469 podStartE2EDuration="9.648956469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:11.646744122 +0000 UTC m=+819.288650407" watchObservedRunningTime="2026-11-04 01:17:11.648956469 +0000 UTC m=+819.290862720"
Nov 04 01:17:11.701522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.701162    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss" event={"ID":"440d2009-5efa-45d4-b976-574d70d33054","Type":"ContainerStarted","Data":"8413b494c8a2dfe7753b4ecff53745d352baad7a838405c1c41e5b0d71555b85"}
Nov 04 01:17:11.701522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.701366    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:11.781485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.781400    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"]
Nov 04 01:17:11.905981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.905872    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:11.905981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:11.905925    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:12.339107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.339075    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:12.339840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.339819    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:12.339886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.339862    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:12.701011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.695787    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:12.755065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.741891    5797 generic.go:356] "Generic (PLEG): container finished" podID="9c1a3431-f7b6-4455-8877-07ce2a668170" containerID="6919b84e26580a64da23a664e962203d49fc67a8f4cc2a17e0136b780fc411de" exitCode=0
Nov 04 01:17:12.755065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.742071    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz" event={"ID":"9c1a3431-f7b6-4455-8877-07ce2a668170","Type":"ContainerDied","Data":"6919b84e26580a64da23a664e962203d49fc67a8f4cc2a17e0136b780fc411de"}
Nov 04 01:17:12.755065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.745393    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss" event={"ID":"440d2009-5efa-45d4-b976-574d70d33054","Type":"ContainerStarted","Data":"0378315dbee705c60b46ad1e3b849e8b79287b4c5841adbdd577298da1a6dc3c"}
Nov 04 01:17:12.824034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.822054    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl" event={"ID":"8d9e5234-5778-4c6b-8747-cdabefef5d3c","Type":"ContainerStarted","Data":"da31062d1c506258f98f5bd48c0b2d689cc5ed9753774374a5888420e41f17d9"}
Nov 04 01:17:12.824034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.822115    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl" event={"ID":"8d9e5234-5778-4c6b-8747-cdabefef5d3c","Type":"ContainerStarted","Data":"574d993b165514f94e3c00391e04cd48745f7eff807285dee1fbc8b5873a4b87"}
Nov 04 01:17:12.845895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.845844    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss" podStartSLOduration=7.845829256 podStartE2EDuration="7.845829256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:12.841331675 +0000 UTC m=+820.483237954" watchObservedRunningTime="2026-11-04 01:17:12.845829256 +0000 UTC m=+820.487735505"
Nov 04 01:17:12.876014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.872397    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl" podStartSLOduration=4.872360374 podStartE2EDuration="4.872360374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:12.871256767 +0000 UTC m=+820.513163050" watchObservedRunningTime="2026-11-04 01:17:12.872360374 +0000 UTC m=+820.514266628"
Nov 04 01:17:12.901390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.900690    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:12.901390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:12.900749    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:13.339543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.339111    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:13.339543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.339169    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:13.644338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.644303    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:13.831257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.831226    5797 generic.go:356] "Generic (PLEG): container finished" podID="440d2009-5efa-45d4-b976-574d70d33054" containerID="0378315dbee705c60b46ad1e3b849e8b79287b4c5841adbdd577298da1a6dc3c" exitCode=0
Nov 04 01:17:13.831483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.831303    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss" event={"ID":"440d2009-5efa-45d4-b976-574d70d33054","Type":"ContainerDied","Data":"0378315dbee705c60b46ad1e3b849e8b79287b4c5841adbdd577298da1a6dc3c"}
Nov 04 01:17:13.832847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.832829    5797 generic.go:356] "Generic (PLEG): container finished" podID="8d9e5234-5778-4c6b-8747-cdabefef5d3c" containerID="da31062d1c506258f98f5bd48c0b2d689cc5ed9753774374a5888420e41f17d9" exitCode=0
Nov 04 01:17:13.833101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.833085    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl" event={"ID":"8d9e5234-5778-4c6b-8747-cdabefef5d3c","Type":"ContainerDied","Data":"da31062d1c506258f98f5bd48c0b2d689cc5ed9753774374a5888420e41f17d9"}
Nov 04 01:17:13.901539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.901451    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:13.901539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:13.901507    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:14.230738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.230712    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:14.278013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.277437    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca\") pod \"9c1a3431-f7b6-4455-8877-07ce2a668170\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") "
Nov 04 01:17:14.278013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.277543    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb\" (UniqueName: \"kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb\") pod \"9c1a3431-f7b6-4455-8877-07ce2a668170\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") "
Nov 04 01:17:14.278013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.277578    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca\") pod \"9c1a3431-f7b6-4455-8877-07ce2a668170\" (UID: \"9c1a3431-f7b6-4455-8877-07ce2a668170\") "
Nov 04 01:17:14.280908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.279313    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca" pod "9c1a3431-f7b6-4455-8877-07ce2a668170" (UID: "9c1a3431-f7b6-4455-8877-07ce2a668170"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:14.311693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.311657    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca" pod "9c1a3431-f7b6-4455-8877-07ce2a668170" (UID: "9c1a3431-f7b6-4455-8877-07ce2a668170"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:14.314167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.314137    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb" pod "9c1a3431-f7b6-4455-8877-07ce2a668170" (UID: "9c1a3431-f7b6-4455-8877-07ce2a668170"). InnerVolumeSpecName "kube-api-access-lj4nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:14.338603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.338577    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:14.338716 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.338636    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:14.406103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.382207    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4nb\" (UniqueName: \"kubernetes.io/projected/9c1a3431-f7b6-4455-8877-07ce2a668170-kube-api-access-lj4nb\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:14.406594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.406104    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:14.406594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.406138    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a3431-f7b6-4455-8877-07ce2a668170-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:14.619012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.612460    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:14.857062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.853904    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"
Nov 04 01:17:14.879740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.867733    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz" event={"ID":"9c1a3431-f7b6-4455-8877-07ce2a668170","Type":"ContainerDied","Data":"7a8f70330c1a2f7b5baf8b5e7c63df272744b3617b49c42cd6419a616b3749c2"}
Nov 04 01:17:14.879740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.867797    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8f70330c1a2f7b5baf8b5e7c63df272744b3617b49c42cd6419a616b3749c2"
Nov 04 01:17:14.913294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.902387    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:14.913294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:14.902457    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:15.327808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.327776    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:15.344008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.343374    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:15.344008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.343429    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:15.416776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.416737    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca\") pod \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") "
Nov 04 01:17:15.417260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.416915    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca\") pod \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") "
Nov 04 01:17:15.417260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.417140    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7\" (UniqueName: \"kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7\") pod \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\" (UID: \"8d9e5234-5778-4c6b-8747-cdabefef5d3c\") "
Nov 04 01:17:15.426359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.422504    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca" pod "8d9e5234-5778-4c6b-8747-cdabefef5d3c" (UID: "8d9e5234-5778-4c6b-8747-cdabefef5d3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:15.426359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.423248    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca" pod "8d9e5234-5778-4c6b-8747-cdabefef5d3c" (UID: "8d9e5234-5778-4c6b-8747-cdabefef5d3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:15.446042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.445928    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7" pod "8d9e5234-5778-4c6b-8747-cdabefef5d3c" (UID: "8d9e5234-5778-4c6b-8747-cdabefef5d3c"). InnerVolumeSpecName "kube-api-access-6lph7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:15.481013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.475210    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:15.542110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.532776    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lph7\" (UniqueName: \"kubernetes.io/projected/8d9e5234-5778-4c6b-8747-cdabefef5d3c-kube-api-access-6lph7\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:15.542110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.532809    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:15.542110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.532825    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9e5234-5778-4c6b-8747-cdabefef5d3c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:15.652153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.651482    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n\" (UniqueName: \"kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n\") pod \"440d2009-5efa-45d4-b976-574d70d33054\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") "
Nov 04 01:17:15.652153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.651750    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca\") pod \"440d2009-5efa-45d4-b976-574d70d33054\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") "
Nov 04 01:17:15.652153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.651872    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca\") pod \"440d2009-5efa-45d4-b976-574d70d33054\" (UID: \"440d2009-5efa-45d4-b976-574d70d33054\") "
Nov 04 01:17:15.666017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.656754    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:15.691325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.691274    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca" pod "440d2009-5efa-45d4-b976-574d70d33054" (UID: "440d2009-5efa-45d4-b976-574d70d33054"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:15.737209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.737163    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca" pod "440d2009-5efa-45d4-b976-574d70d33054" (UID: "440d2009-5efa-45d4-b976-574d70d33054"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:15.738170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.738122    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n" pod "440d2009-5efa-45d4-b976-574d70d33054" (UID: "440d2009-5efa-45d4-b976-574d70d33054"). InnerVolumeSpecName "kube-api-access-pd87n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:15.780014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.770191    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440d2009-5efa-45d4-b976-574d70d33054-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:15.780014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.770231    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440d2009-5efa-45d4-b976-574d70d33054-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:15.780014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.770250    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd87n\" (UniqueName: \"kubernetes.io/projected/440d2009-5efa-45d4-b976-574d70d33054-kube-api-access-pd87n\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:15.908013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.906490    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:15.908013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.906548    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:15.917976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.917951    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"
Nov 04 01:17:15.919015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.917965    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss" event={"ID":"440d2009-5efa-45d4-b976-574d70d33054","Type":"ContainerDied","Data":"8413b494c8a2dfe7753b4ecff53745d352baad7a838405c1c41e5b0d71555b85"}
Nov 04 01:17:15.919015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.918673    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8413b494c8a2dfe7753b4ecff53745d352baad7a838405c1c41e5b0d71555b85"
Nov 04 01:17:15.924159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.924129    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl" event={"ID":"8d9e5234-5778-4c6b-8747-cdabefef5d3c","Type":"ContainerDied","Data":"574d993b165514f94e3c00391e04cd48745f7eff807285dee1fbc8b5873a4b87"}
Nov 04 01:17:15.924301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.924181    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574d993b165514f94e3c00391e04cd48745f7eff807285dee1fbc8b5873a4b87"
Nov 04 01:17:15.924301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:15.924211    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"
Nov 04 01:17:16.110114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.110077    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"]
Nov 04 01:17:16.111318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111298    5797 cpu_manager.go:414] "containerMap: removing container" podUID="440d2009-5efa-45d4-b976-574d70d33054" containerName="revoke-keys"
Nov 04 01:17:16.111455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111442    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440d2009-5efa-45d4-b976-574d70d33054" containerName="revoke-keys"
Nov 04 01:17:16.111537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111528    5797 cpu_manager.go:414] "containerMap: removing container" podUID="9c1a3431-f7b6-4455-8877-07ce2a668170" containerName="revoke-keys"
Nov 04 01:17:16.111605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111596    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1a3431-f7b6-4455-8877-07ce2a668170" containerName="revoke-keys"
Nov 04 01:17:16.111700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111692    5797 cpu_manager.go:414] "containerMap: removing container" podUID="8d9e5234-5778-4c6b-8747-cdabefef5d3c" containerName="revoke-keys"
Nov 04 01:17:16.111763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111754    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9e5234-5778-4c6b-8747-cdabefef5d3c" containerName="revoke-keys"
Nov 04 01:17:16.112024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.111980    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="440d2009-5efa-45d4-b976-574d70d33054" containerName="revoke-keys"
Nov 04 01:17:16.112114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.112105    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1a3431-f7b6-4455-8877-07ce2a668170" containerName="revoke-keys"
Nov 04 01:17:16.112198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.112190    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9e5234-5778-4c6b-8747-cdabefef5d3c" containerName="revoke-keys"
Nov 04 01:17:16.116509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.116489    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"]
Nov 04 01:17:16.116760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.116712    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.119213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.119192    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.119376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.119359    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.119579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.119558    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mccw\" (UniqueName: \"kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.120611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.120592    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:17:16.120774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.120762    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:17:16.221532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.221433    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.221532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.221490    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.221775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.221617    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mccw\" (UniqueName: \"kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.222799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.222774    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.225143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.225121    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.235272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.235253    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mccw\" (UniqueName: \"kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw\") pod \"maas-api-revoke-sub-7418de00-qklml\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.338707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.338672    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:16.339008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.338962    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:16.435393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.435361    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:16.826133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.825252    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:16.830041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.829296    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"]
Nov 04 01:17:16.910208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.910170    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 04 01:17:16.910446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.910423    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 04 01:17:16.914424 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:16.914374    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc488997b_e965_4fcf_a24f_a329a790b438.slice/crio-5d6b36fc7e32f7bb7943b0ee3587919aa484813cd81e9dae015d8f77ecfd1e30 WatchSource:0}: Error finding container 5d6b36fc7e32f7bb7943b0ee3587919aa484813cd81e9dae015d8f77ecfd1e30: Status 404 returned error can't find the container with id 5d6b36fc7e32f7bb7943b0ee3587919aa484813cd81e9dae015d8f77ecfd1e30
Nov 04 01:17:16.952390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:16.952349    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml" event={"ID":"c488997b-e965-4fcf-a24f-a329a790b438","Type":"ContainerStarted","Data":"5d6b36fc7e32f7bb7943b0ee3587919aa484813cd81e9dae015d8f77ecfd1e30"}
Nov 04 01:17:17.339405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.339367    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:17.339581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.339441    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:17.724464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.724387    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:17.793610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.793418    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"]
Nov 04 01:17:17.802053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.800133    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.821013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.817590    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"]
Nov 04 01:17:17.873851 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.873817    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdj9z\" (UniqueName: \"kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.874593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.874560    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.874946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.874927    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.900878 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.900829    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:17.968012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.964786    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml" event={"ID":"c488997b-e965-4fcf-a24f-a329a790b438","Type":"ContainerStarted","Data":"479b0dc99c87b2c71dfeb8125b5ae9195bc77b651a6997bbb8d78fcfb3ce39f7"}
Nov 04 01:17:17.977135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.977038    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdj9z\" (UniqueName: \"kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.977135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.977086    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.977356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.977303    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.977946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.977915    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.980291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.980246    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:17.985340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.985250    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml" podStartSLOduration=1.985202107 podStartE2EDuration="1.985202107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:17.984485162 +0000 UTC m=+825.626391449" watchObservedRunningTime="2026-11-04 01:17:17.985202107 +0000 UTC m=+825.627108366"
Nov 04 01:17:17.992743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:17.992692    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdj9z\" (UniqueName: \"kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z\") pod \"maas-api-revoke-sub-5620c6bb-9875l\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:18.135602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.135562    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:18.342570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.342039    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 04 01:17:18.342570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.342116    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 04 01:17:18.372052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.370365    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:18.372052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.371929    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:17:18.440810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.440775    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"]
Nov 04 01:17:18.693613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.693578    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:18.976597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.976499    5797 generic.go:356] "Generic (PLEG): container finished" podID="c488997b-e965-4fcf-a24f-a329a790b438" containerID="479b0dc99c87b2c71dfeb8125b5ae9195bc77b651a6997bbb8d78fcfb3ce39f7" exitCode=0
Nov 04 01:17:18.977025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.976587    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml" event={"ID":"c488997b-e965-4fcf-a24f-a329a790b438","Type":"ContainerDied","Data":"479b0dc99c87b2c71dfeb8125b5ae9195bc77b651a6997bbb8d78fcfb3ce39f7"}
Nov 04 01:17:18.978465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.978440    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l" event={"ID":"2843f357-e6c6-456c-8e72-965e60ec35f7","Type":"ContainerStarted","Data":"21935c3a12d86eddea14a460bbc320be97830d0c42ab1d83d5d8e9c6b1e46b75"}
Nov 04 01:17:18.978597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:18.978490    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l" event={"ID":"2843f357-e6c6-456c-8e72-965e60ec35f7","Type":"ContainerStarted","Data":"bd7035cbaf2787bd531e7ec1b06fd1a8034b1eb73d15f54130474433da0a4b91"}
Nov 04 01:17:19.017664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.017611    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l" podStartSLOduration=2.01759833 podStartE2EDuration="2.01759833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:19.015477131 +0000 UTC m=+826.657383418" watchObservedRunningTime="2026-11-04 01:17:19.01759833 +0000 UTC m=+826.659504579"
Nov 04 01:17:19.343054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.342966    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:19.343462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.343444    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:19.344302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.344284    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:17:19.500791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.500761    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:19.987975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.987943    5797 generic.go:356] "Generic (PLEG): container finished" podID="2843f357-e6c6-456c-8e72-965e60ec35f7" containerID="21935c3a12d86eddea14a460bbc320be97830d0c42ab1d83d5d8e9c6b1e46b75" exitCode=0
Nov 04 01:17:19.988444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:19.988024    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l" event={"ID":"2843f357-e6c6-456c-8e72-965e60ec35f7","Type":"ContainerDied","Data":"21935c3a12d86eddea14a460bbc320be97830d0c42ab1d83d5d8e9c6b1e46b75"}
Nov 04 01:17:20.186743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.186715    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:20.366887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.366856    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca\") pod \"c488997b-e965-4fcf-a24f-a329a790b438\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") "
Nov 04 01:17:20.367096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.367019    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca\") pod \"c488997b-e965-4fcf-a24f-a329a790b438\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") "
Nov 04 01:17:20.367096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.367087    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw\" (UniqueName: \"kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw\") pod \"c488997b-e965-4fcf-a24f-a329a790b438\" (UID: \"c488997b-e965-4fcf-a24f-a329a790b438\") "
Nov 04 01:17:20.367370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.367348    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca" pod "c488997b-e965-4fcf-a24f-a329a790b438" (UID: "c488997b-e965-4fcf-a24f-a329a790b438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:20.367710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.367693    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:20.369076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.369049    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca" pod "c488997b-e965-4fcf-a24f-a329a790b438" (UID: "c488997b-e965-4fcf-a24f-a329a790b438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:20.369239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.369214    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw" pod "c488997b-e965-4fcf-a24f-a329a790b438" (UID: "c488997b-e965-4fcf-a24f-a329a790b438"). InnerVolumeSpecName "kube-api-access-5mccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:20.469675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.469637    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c488997b-e965-4fcf-a24f-a329a790b438-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:20.469675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.469669    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mccw\" (UniqueName: \"kubernetes.io/projected/c488997b-e965-4fcf-a24f-a329a790b438-kube-api-access-5mccw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:20.598895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.598858    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:20.997490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.997456    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"
Nov 04 01:17:20.997968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.997487    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml" event={"ID":"c488997b-e965-4fcf-a24f-a329a790b438","Type":"ContainerDied","Data":"5d6b36fc7e32f7bb7943b0ee3587919aa484813cd81e9dae015d8f77ecfd1e30"}
Nov 04 01:17:20.997968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:20.997560    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6b36fc7e32f7bb7943b0ee3587919aa484813cd81e9dae015d8f77ecfd1e30"
Nov 04 01:17:21.197980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.197953    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:21.236899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.236868    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca\") pod \"2843f357-e6c6-456c-8e72-965e60ec35f7\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") "
Nov 04 01:17:21.237104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.236953    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z\" (UniqueName: \"kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z\") pod \"2843f357-e6c6-456c-8e72-965e60ec35f7\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") "
Nov 04 01:17:21.237266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.237243    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca\") pod \"2843f357-e6c6-456c-8e72-965e60ec35f7\" (UID: \"2843f357-e6c6-456c-8e72-965e60ec35f7\") "
Nov 04 01:17:21.237664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.237638    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca" pod "2843f357-e6c6-456c-8e72-965e60ec35f7" (UID: "2843f357-e6c6-456c-8e72-965e60ec35f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:21.237827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.237808    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:21.239284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.239259    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z" pod "2843f357-e6c6-456c-8e72-965e60ec35f7" (UID: "2843f357-e6c6-456c-8e72-965e60ec35f7"). InnerVolumeSpecName "kube-api-access-fdj9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:21.239558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.239534    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca" pod "2843f357-e6c6-456c-8e72-965e60ec35f7" (UID: "2843f357-e6c6-456c-8e72-965e60ec35f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:21.339575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.339537    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdj9z\" (UniqueName: \"kubernetes.io/projected/2843f357-e6c6-456c-8e72-965e60ec35f7-kube-api-access-fdj9z\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:21.339575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.339578    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2843f357-e6c6-456c-8e72-965e60ec35f7-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:21.497103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:21.497073    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:22.005830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.005797    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"
Nov 04 01:17:22.006242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.005836    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l" event={"ID":"2843f357-e6c6-456c-8e72-965e60ec35f7","Type":"ContainerDied","Data":"bd7035cbaf2787bd531e7ec1b06fd1a8034b1eb73d15f54130474433da0a4b91"}
Nov 04 01:17:22.006242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.005891    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7035cbaf2787bd531e7ec1b06fd1a8034b1eb73d15f54130474433da0a4b91"
Nov 04 01:17:22.303749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.303669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:22.806780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.806742    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"]
Nov 04 01:17:22.807417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.807403    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2843f357-e6c6-456c-8e72-965e60ec35f7" containerName="revoke-keys"
Nov 04 01:17:22.807495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.807418    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2843f357-e6c6-456c-8e72-965e60ec35f7" containerName="revoke-keys"
Nov 04 01:17:22.807495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.807434    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c488997b-e965-4fcf-a24f-a329a790b438" containerName="revoke-keys"
Nov 04 01:17:22.807495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.807439    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c488997b-e965-4fcf-a24f-a329a790b438" containerName="revoke-keys"
Nov 04 01:17:22.807588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.807561    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2843f357-e6c6-456c-8e72-965e60ec35f7" containerName="revoke-keys"
Nov 04 01:17:22.807588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.807569    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c488997b-e965-4fcf-a24f-a329a790b438" containerName="revoke-keys"
Nov 04 01:17:22.811668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.811642    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.815444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.815422    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:17:22.815600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.815580    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:17:22.820615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.820588    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"]
Nov 04 01:17:22.865923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.865891    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.866189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.866097    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.866506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.866484    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fll8j\" (UniqueName: \"kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.968509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.968472    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.968833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.968795    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fll8j\" (UniqueName: \"kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.969267 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.969246    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.969376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.969358    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.972423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.972398    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:22.981855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:22.981826    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fll8j\" (UniqueName: \"kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j\") pod \"maas-api-revoke-sub-3ea9851d-hgrg6\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:23.094438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:23.094324    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:23.131270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:23.131233    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:23.426338 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:23.426231    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1adcbf_a7de_4d0e_a138_2c3b7ce0129b.slice/crio-b335befe9797f32c918a204b021d03360bf38bd26682cb02692130e055eb093c WatchSource:0}: Error finding container b335befe9797f32c918a204b021d03360bf38bd26682cb02692130e055eb093c: Status 404 returned error can't find the container with id b335befe9797f32c918a204b021d03360bf38bd26682cb02692130e055eb093c
Nov 04 01:17:23.427730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:23.427698    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"]
Nov 04 01:17:24.031103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:24.031066    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:24.038410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:24.038350    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6" event={"ID":"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b","Type":"ContainerStarted","Data":"f3ea11000c609ca51945a9d51a1e898b96c2fe53bd2363a0588025db42b7ac1f"}
Nov 04 01:17:24.038563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:24.038450    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6" event={"ID":"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b","Type":"ContainerStarted","Data":"b335befe9797f32c918a204b021d03360bf38bd26682cb02692130e055eb093c"}
Nov 04 01:17:24.061225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:24.061170    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6" podStartSLOduration=2.061157353 podStartE2EDuration="2.061157353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:24.058057005 +0000 UTC m=+831.699963279" watchObservedRunningTime="2026-11-04 01:17:24.061157353 +0000 UTC m=+831.703063602"
Nov 04 01:17:24.996202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:24.996156    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:25.051109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:25.051046    5797 generic.go:356] "Generic (PLEG): container finished" podID="da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" containerID="f3ea11000c609ca51945a9d51a1e898b96c2fe53bd2363a0588025db42b7ac1f" exitCode=0
Nov 04 01:17:25.051321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:25.051145    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6" event={"ID":"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b","Type":"ContainerDied","Data":"f3ea11000c609ca51945a9d51a1e898b96c2fe53bd2363a0588025db42b7ac1f"}
Nov 04 01:17:25.886534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:25.886499    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:26.227850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.227816    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:26.258244 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:26.258212    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:17:26.345094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.345065    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca\") pod \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") "
Nov 04 01:17:26.345266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.345214    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j\" (UniqueName: \"kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j\") pod \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") "
Nov 04 01:17:26.345330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.345273    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca\") pod \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\" (UID: \"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b\") "
Nov 04 01:17:26.345664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.345636    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca" pod "da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" (UID: "da1adcbf-a7de-4d0e-a138-2c3b7ce0129b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:26.345870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.345850    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:26.347342 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.347316    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca" pod "da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" (UID: "da1adcbf-a7de-4d0e-a138-2c3b7ce0129b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:26.347451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.347403    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j" pod "da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" (UID: "da1adcbf-a7de-4d0e-a138-2c3b7ce0129b"). InnerVolumeSpecName "kube-api-access-fll8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:26.447673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.447593    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fll8j\" (UniqueName: \"kubernetes.io/projected/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-kube-api-access-fll8j\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:26.447673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.447626    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:26.719068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:26.718340    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:27.070471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:27.070397    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"
Nov 04 01:17:27.070471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:27.070425    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6" event={"ID":"da1adcbf-a7de-4d0e-a138-2c3b7ce0129b","Type":"ContainerDied","Data":"b335befe9797f32c918a204b021d03360bf38bd26682cb02692130e055eb093c"}
Nov 04 01:17:27.070471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:27.070471    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b335befe9797f32c918a204b021d03360bf38bd26682cb02692130e055eb093c"
Nov 04 01:17:27.489435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:27.489129    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:28.437923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:28.437884    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:29.283439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.283401    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:29.434435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.434399    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"]
Nov 04 01:17:29.435244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.435223    5797 cpu_manager.go:414] "containerMap: removing container" podUID="da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" containerName="revoke-keys"
Nov 04 01:17:29.435244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.435243    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" containerName="revoke-keys"
Nov 04 01:17:29.435445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.435432    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" containerName="revoke-keys"
Nov 04 01:17:29.439052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.439034    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.450718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.445698    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:17:29.450718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.445772    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:17:29.492791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.492758    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"]
Nov 04 01:17:29.511079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.511042    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.511256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.511096    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.511339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.511290    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwft\" (UniqueName: \"kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.615775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.615725    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.615966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.615787    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.616115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.615972    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwft\" (UniqueName: \"kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.617503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.617372    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.619650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.619619    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.630133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.630108    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwft\" (UniqueName: \"kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft\") pod \"maas-api-revoke-sub-c81529f6-8gd8f\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.795308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.795277    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:29.896572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.896539    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"]
Nov 04 01:17:29.911546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.911501    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"]
Nov 04 01:17:29.911825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.911616    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:29.921760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.920941    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9kj\" (UniqueName: \"kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:29.921760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.921193    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:29.921760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:29.921246    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.022830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.022745    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9kj\" (UniqueName: \"kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.023029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.022963    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.023029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.023025    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.023880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.023844    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.025739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.025712    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.031597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.031557    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:30.037084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.037061    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9kj\" (UniqueName: \"kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj\") pod \"maas-api-revoke-sub-0b4b84d3-2fntb\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.094818 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:30.094787    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f8ec37_fd43_4212_97ae_358737c09e33.slice/crio-7c49da2c1ac464eb497e6312da34ce24746a4a28c3cc50c7cdecb1e157913c7d WatchSource:0}: Error finding container 7c49da2c1ac464eb497e6312da34ce24746a4a28c3cc50c7cdecb1e157913c7d: Status 404 returned error can't find the container with id 7c49da2c1ac464eb497e6312da34ce24746a4a28c3cc50c7cdecb1e157913c7d
Nov 04 01:17:30.096194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.096168    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"]
Nov 04 01:17:30.232008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.231962    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:30.232183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.232068    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:30.524949 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:30.524920    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3107ff_b7e8_4895_a996_3846ea3b7f8d.slice/crio-5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef WatchSource:0}: Error finding container 5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef: Status 404 returned error can't find the container with id 5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef
Nov 04 01:17:30.525342 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.525051    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"]
Nov 04 01:17:30.980308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:30.980273    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:31.107220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.107180    5797 generic.go:356] "Generic (PLEG): container finished" podID="20f8ec37-fd43-4212-97ae-358737c09e33" containerID="09bece2c3294c6c4755a7e13918f863d606b7ddef18985285e263956f65bba71" exitCode=0
Nov 04 01:17:31.107389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.107263    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f" event={"ID":"20f8ec37-fd43-4212-97ae-358737c09e33","Type":"ContainerDied","Data":"09bece2c3294c6c4755a7e13918f863d606b7ddef18985285e263956f65bba71"}
Nov 04 01:17:31.107389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.107341    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f" event={"ID":"20f8ec37-fd43-4212-97ae-358737c09e33","Type":"ContainerStarted","Data":"7c49da2c1ac464eb497e6312da34ce24746a4a28c3cc50c7cdecb1e157913c7d"}
Nov 04 01:17:31.109004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.108970    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb" event={"ID":"fc3107ff-b7e8-4895-a996-3846ea3b7f8d","Type":"ContainerStarted","Data":"c76c2bae4055926baf8c01297fbbbe5755444f65db979642e69ed299795c2c7c"}
Nov 04 01:17:31.109111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.109025    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb" event={"ID":"fc3107ff-b7e8-4895-a996-3846ea3b7f8d","Type":"ContainerStarted","Data":"5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef"}
Nov 04 01:17:31.148953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.148911    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb" podStartSLOduration=2.148899279 podStartE2EDuration="2.148899279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:31.144186276 +0000 UTC m=+838.786092539" watchObservedRunningTime="2026-11-04 01:17:31.148899279 +0000 UTC m=+838.790805554"
Nov 04 01:17:31.280102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.280027    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:31.701346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.701299    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:31.854551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.854467    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"]
Nov 04 01:17:31.859188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.859164    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.867223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.867198    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"]
Nov 04 01:17:31.895809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.895773    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgdq\" (UniqueName: \"kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.895966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.895832    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.896072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.896055    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.997848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.997811    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.998046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.997950    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgdq\" (UniqueName: \"kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.998099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.998064    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:31.998751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:31.998728    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:32.000283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.000263    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:32.010074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.010054    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgdq\" (UniqueName: \"kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq\") pod \"maas-api-revoke-sub-57366fe0-wd22h\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:32.122834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.122801    5797 generic.go:356] "Generic (PLEG): container finished" podID="fc3107ff-b7e8-4895-a996-3846ea3b7f8d" containerID="c76c2bae4055926baf8c01297fbbbe5755444f65db979642e69ed299795c2c7c" exitCode=0
Nov 04 01:17:32.123046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.122894    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb" event={"ID":"fc3107ff-b7e8-4895-a996-3846ea3b7f8d","Type":"ContainerDied","Data":"c76c2bae4055926baf8c01297fbbbe5755444f65db979642e69ed299795c2c7c"}
Nov 04 01:17:32.178236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.178203    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:32.286759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.286393    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:32.417223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.417140    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca\") pod \"20f8ec37-fd43-4212-97ae-358737c09e33\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") "
Nov 04 01:17:32.417398 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.417276    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca\") pod \"20f8ec37-fd43-4212-97ae-358737c09e33\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") "
Nov 04 01:17:32.417398 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.417322    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft\" (UniqueName: \"kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft\") pod \"20f8ec37-fd43-4212-97ae-358737c09e33\" (UID: \"20f8ec37-fd43-4212-97ae-358737c09e33\") "
Nov 04 01:17:32.417645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.417622    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca" pod "20f8ec37-fd43-4212-97ae-358737c09e33" (UID: "20f8ec37-fd43-4212-97ae-358737c09e33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:32.418064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.418039    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:32.419441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.419413    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca" pod "20f8ec37-fd43-4212-97ae-358737c09e33" (UID: "20f8ec37-fd43-4212-97ae-358737c09e33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:32.419848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.419821    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft" pod "20f8ec37-fd43-4212-97ae-358737c09e33" (UID: "20f8ec37-fd43-4212-97ae-358737c09e33"). InnerVolumeSpecName "kube-api-access-gzwft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:32.519414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.519388    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzwft\" (UniqueName: \"kubernetes.io/projected/20f8ec37-fd43-4212-97ae-358737c09e33-kube-api-access-gzwft\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:32.519490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.519417    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f8ec37-fd43-4212-97ae-358737c09e33-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:32.521247 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:32.521218    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc128a7_238a_4041_b3f8_a5ad95d97ee1.slice/crio-afd80f817f854021b336cebd365dd0f7c576879c62a7f5c942be470287394a21 WatchSource:0}: Error finding container afd80f817f854021b336cebd365dd0f7c576879c62a7f5c942be470287394a21: Status 404 returned error can't find the container with id afd80f817f854021b336cebd365dd0f7c576879c62a7f5c942be470287394a21
Nov 04 01:17:32.522850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.522828    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"]
Nov 04 01:17:32.552039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.552013    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"]
Nov 04 01:17:32.552663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.552651    5797 cpu_manager.go:414] "containerMap: removing container" podUID="20f8ec37-fd43-4212-97ae-358737c09e33" containerName="revoke-keys"
Nov 04 01:17:32.552722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.552665    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f8ec37-fd43-4212-97ae-358737c09e33" containerName="revoke-keys"
Nov 04 01:17:32.552836 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.552827    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f8ec37-fd43-4212-97ae-358737c09e33" containerName="revoke-keys"
Nov 04 01:17:32.558020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.558003    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.571983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.571942    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"]
Nov 04 01:17:32.601434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.601403    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"]
Nov 04 01:17:32.620962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.620920    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.620971    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621020    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621096    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621167    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621277    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621333    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621361    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.621552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.621450    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnc6\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.834063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.833074    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"]
Nov 04 01:17:32.834063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.833220    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849280    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849335    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849441    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnc6\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849597    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849626    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849662    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849750    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849833    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.849971    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.869062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.850587    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.904505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.893750    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.904505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.894034    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.904505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.894650    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.904505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.896296    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.906376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.906332    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:32.910374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:32.910143    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:33.027040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.007141    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnc6\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:33.048329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.047276    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.083982    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084070    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084132    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084196    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084240    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtq6\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084304    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084395    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084467    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.085009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.084563    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.194015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.191299    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214324    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214376    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214427    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214479    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214517    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtq6\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214574    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214655    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214763    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.215014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.214856    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.228054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.226046    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.263093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.256004    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.278053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.263467    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.278053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.275406    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.278053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.275683    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.278053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.275896    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.315478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.315338    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h" event={"ID":"0fc128a7-238a-4041-b3f8-a5ad95d97ee1","Type":"ContainerStarted","Data":"afd80f817f854021b336cebd365dd0f7c576879c62a7f5c942be470287394a21"}
Nov 04 01:17:33.336264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.325649    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.365233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.355411    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"
Nov 04 01:17:33.365233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.356772    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f" event={"ID":"20f8ec37-fd43-4212-97ae-358737c09e33","Type":"ContainerDied","Data":"7c49da2c1ac464eb497e6312da34ce24746a4a28c3cc50c7cdecb1e157913c7d"}
Nov 04 01:17:33.365233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.356844    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c49da2c1ac464eb497e6312da34ce24746a4a28c3cc50c7cdecb1e157913c7d"
Nov 04 01:17:33.426015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.393964    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtq6\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.464026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.458098    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.671020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.669537    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:33.858073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:33.843284    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:34.458059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:34.450954    5797 generic.go:356] "Generic (PLEG): container finished" podID="0fc128a7-238a-4041-b3f8-a5ad95d97ee1" containerID="4a54c15163995711a617173e3afdae3a948ad2a710c7175f0b1152a9491651d8" exitCode=0
Nov 04 01:17:34.458059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:34.451208    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h" event={"ID":"0fc128a7-238a-4041-b3f8-a5ad95d97ee1","Type":"ContainerDied","Data":"4a54c15163995711a617173e3afdae3a948ad2a710c7175f0b1152a9491651d8"}
Nov 04 01:17:34.483720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:34.481960    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:34.679192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:34.679117    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:34.752576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:34.752466    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:34.753587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:34.753552    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.087196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.087093    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.504021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.490812    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.505759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.505730    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:17:35.537073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.536957    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.885922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.546205    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.886321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.597494    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.887886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.691697    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:35.914662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.856794    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:35.914662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.907896    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:35.914662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.908007    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:35.914662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.908061    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:35.917163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.917134    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:17:35.917381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.917365    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:17:35.918541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.880447    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:35.924017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.923853    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:17:36.001086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:35.985230    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"]
Nov 04 01:17:36.157020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.120059    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:36.200014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.180061    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:17:36.531015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.505805    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:36.690450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.689217    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:36.945223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.945103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:37.308072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.955240    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:37.308555 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.955300    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:37.310236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:36.955343    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:37.377199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.078320    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:37.386012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.078381    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:37.411017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.122190    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca\") pod \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") "
Nov 04 01:17:37.411017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.410657    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj\" (UniqueName: \"kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj\") pod \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") "
Nov 04 01:17:37.411017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.410803    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca\") pod \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\" (UID: \"fc3107ff-b7e8-4895-a996-3846ea3b7f8d\") "
Nov 04 01:17:37.439043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.123076    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca" pod "fc3107ff-b7e8-4895-a996-3846ea3b7f8d" (UID: "fc3107ff-b7e8-4895-a996-3846ea3b7f8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:37.463146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.251539    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:37.502742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.484115    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:37.579877 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:37.521484    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a85dd86_0f39_49f1_ac8e_0986ee8962b8.slice/crio-8deb9516fd4121711f2ef9a44dab32a9929b067c263fe092ee48d2591e108ae5 WatchSource:0}: Error finding container 8deb9516fd4121711f2ef9a44dab32a9929b067c263fe092ee48d2591e108ae5: Status 404 returned error can't find the container with id 8deb9516fd4121711f2ef9a44dab32a9929b067c263fe092ee48d2591e108ae5
Nov 04 01:17:37.752628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:37.666443    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj" pod "fc3107ff-b7e8-4895-a996-3846ea3b7f8d" (UID: "fc3107ff-b7e8-4895-a996-3846ea3b7f8d"). InnerVolumeSpecName "kube-api-access-pr9kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:38.091956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.091908    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.146627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.146592    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca" pod "fc3107ff-b7e8-4895-a996-3846ea3b7f8d" (UID: "fc3107ff-b7e8-4895-a996-3846ea3b7f8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:38.349583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.349541    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:17:38.350462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.350431    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:38.350922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.350899    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:38.405160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.360084    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:38.418209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.360252    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.418449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.360717    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.418564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.394950    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.418670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.395091    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.418782 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:38.395318    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.878s"
Nov 04 01:17:38.419150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.419127    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:17:38.470763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.461631    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.470763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.462976    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:38.470763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.463053    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:38.505047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.504289    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:38.505047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.504413    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:38.505047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.504462    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:17:38.764124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.764088    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:38.764378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.764353    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr9kj\" (UniqueName: \"kubernetes.io/projected/fc3107ff-b7e8-4895-a996-3846ea3b7f8d-kube-api-access-pr9kj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:38.812015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.795047    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:17:38.812015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:38.795815    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:17:39.132701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.108819    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:39.358445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.358391    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.378582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.358089    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:39.435623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.435575    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.437264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.437229    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.446011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.442075    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.718072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.714611    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.801131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.801072    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.849023 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:39.832200    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.036s"
Nov 04 01:17:39.878031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.869584    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:39.887030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.879253    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"
Nov 04 01:17:39.887030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.879467    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb" event={"ID":"fc3107ff-b7e8-4895-a996-3846ea3b7f8d","Type":"ContainerDied","Data":"5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef"}
Nov 04 01:17:39.887030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.879515    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef"
Nov 04 01:17:39.924960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.924572    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" event={"ID":"d6f64df9-087e-45ed-b182-50ce24b355ba","Type":"ContainerStarted","Data":"0c77d05a6da493ba166640cf0c8ba17f625c3fe458df8aa43850a2d8935c1fee"}
Nov 04 01:17:39.924960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.924639    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" event={"ID":"d6f64df9-087e-45ed-b182-50ce24b355ba","Type":"ContainerStarted","Data":"c5ca5a9a396fb4fbd0ede4002bd499c6b600550cd2973739defefe9d4783acfb"}
Nov 04 01:17:39.946251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:39.946218    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:40.029310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.029269    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq\" (UniqueName: \"kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq\") pod \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") "
Nov 04 01:17:40.030061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.030037    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca\") pod \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") "
Nov 04 01:17:40.030581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.030547    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca\") pod \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\" (UID: \"0fc128a7-238a-4041-b3f8-a5ad95d97ee1\") "
Nov 04 01:17:40.031937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.031423    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca" pod "0fc128a7-238a-4041-b3f8-a5ad95d97ee1" (UID: "0fc128a7-238a-4041-b3f8-a5ad95d97ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:40.037949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.037930    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:40.057522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.057492    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"]
Nov 04 01:17:40.065922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.065887    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca" pod "0fc128a7-238a-4041-b3f8-a5ad95d97ee1" (UID: "0fc128a7-238a-4041-b3f8-a5ad95d97ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:40.181014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.157343    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:40.231081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.230226    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:40.231081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.230307    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:40.295027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.272070    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:40.366621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.365387    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq" pod "0fc128a7-238a-4041-b3f8-a5ad95d97ee1" (UID: "0fc128a7-238a-4041-b3f8-a5ad95d97ee1"). InnerVolumeSpecName "kube-api-access-4pgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:40.366621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.365879    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:40.366621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.365919    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:40.385334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.385230    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:40.393187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.393079    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:40.450045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.440528    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:40.450045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.442919    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgdq\" (UniqueName: \"kubernetes.io/projected/0fc128a7-238a-4041-b3f8-a5ad95d97ee1-kube-api-access-4pgdq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:40.450045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.443260    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:40.493066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.470304    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:40.572016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.570809    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 04 01:17:40.576215 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:40.576174    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3107ff_b7e8_4895_a996_3846ea3b7f8d.slice/crio-5ad6febf118ee849edf087f7e1d6675625c91e31e07e000324bb30f6f412c0ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3107ff_b7e8_4895_a996_3846ea3b7f8d.slice\": RecentStats: unable to find data in memory cache]"
Nov 04 01:17:40.613016 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:40.592416    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3107ff_b7e8_4895_a996_3846ea3b7f8d.slice\": RecentStats: unable to find data in memory cache]"
Nov 04 01:17:40.716017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.715938    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:40.720013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.717770    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"]
Nov 04 01:17:40.744069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.743971    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0fc128a7-238a-4041-b3f8-a5ad95d97ee1" containerName="revoke-keys"
Nov 04 01:17:40.744350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.744330    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc128a7-238a-4041-b3f8-a5ad95d97ee1" containerName="revoke-keys"
Nov 04 01:17:40.744521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.744507    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fc3107ff-b7e8-4895-a996-3846ea3b7f8d" containerName="revoke-keys"
Nov 04 01:17:40.744708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.744694    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3107ff-b7e8-4895-a996-3846ea3b7f8d" containerName="revoke-keys"
Nov 04 01:17:40.749013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.745119    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc128a7-238a-4041-b3f8-a5ad95d97ee1" containerName="revoke-keys"
Nov 04 01:17:40.749567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.749552    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3107ff-b7e8-4895-a996-3846ea3b7f8d" containerName="revoke-keys"
Nov 04 01:17:40.835013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.827771    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:40.835390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.830450    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:40.890217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.874679    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:40.939013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.933731    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:40.950526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.939565    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46jx\" (UniqueName: \"kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:40.951091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:40.951026    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.038645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.032020    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"]
Nov 04 01:17:41.062017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.055734    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.062017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.055874    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.062017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.056055    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46jx\" (UniqueName: \"kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.062017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.057311    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.135009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.131120    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:41.135009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.131195    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.175014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.159305    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:41.175014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.159368    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.175014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.162168    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h" event={"ID":"0fc128a7-238a-4041-b3f8-a5ad95d97ee1","Type":"ContainerDied","Data":"afd80f817f854021b336cebd365dd0f7c576879c62a7f5c942be470287394a21"}
Nov 04 01:17:41.175014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.162235    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd80f817f854021b336cebd365dd0f7c576879c62a7f5c942be470287394a21"
Nov 04 01:17:41.175014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.162306    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"
Nov 04 01:17:41.200017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.193242    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" event={"ID":"0a85dd86-0f39-49f1-ac8e-0986ee8962b8","Type":"ContainerStarted","Data":"46c6dee6e591a6bf6be9bd8987bf1d37e614c76e776fdbf31e611fa4c55e4e49"}
Nov 04 01:17:41.200017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.193310    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" event={"ID":"0a85dd86-0f39-49f1-ac8e-0986ee8962b8","Type":"ContainerStarted","Data":"8deb9516fd4121711f2ef9a44dab32a9929b067c263fe092ee48d2591e108ae5"}
Nov 04 01:17:41.209015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.207652    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.223620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.223555    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:41.224611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.224574    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:41.384078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.378083    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46jx\" (UniqueName: \"kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx\") pod \"maas-api-revoke-sub-c89b2f07-bplc2\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.432464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.432412    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:41.438025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.435204    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.456079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.438582    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.464027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.456639    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.464027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.457814    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:17:41.491152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.491106    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:41.581031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.565209    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:41.581031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.565279    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.698014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.686813    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:41.755181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.725471    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:41.755363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.755221    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:41.768024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.725611    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:41.850564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.850501    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:41.854104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.850833    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:41.948670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.948612    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:42.009046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.970482    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:42.009046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:41.994611    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Nov 04 01:17:42.028042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.027264    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded"
Nov 04 01:17:42.454013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.412414    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:42.516015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.510651    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:42.516015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.484230    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:42.598017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.463911    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:42.598017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.484343    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:42.663730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.484271    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:17:42.747366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.484379    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:42.818765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.818708    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:42.824147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.824109    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:42.853164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.819886    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:42.853164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.846239    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:42.965705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:42.965658    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:43.000206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.000099    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:43.026015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.025965    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:43.026284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.026255    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:43.102142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.102081    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:43.208624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.208539    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podStartSLOduration=11.2085151 podStartE2EDuration="11.2085151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:39.96073835 +0000 UTC m=+847.602644624" watchObservedRunningTime="2026-11-04 01:17:43.2085151 +0000 UTC m=+850.850421357"
Nov 04 01:17:43.507088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.448107    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:43.507088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.448164    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:43.532883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.532848    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:43.610065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.598670    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:43.617304 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:43.617269    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.105s"
Nov 04 01:17:43.617460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.617432    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:17:43.617502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.617481    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:17:43.617534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.617513    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:17:43.617579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.617541    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:17:43.617579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.617572    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:17:43.635564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.635539    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:17:43.642622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.641229    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:17:43.642622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.641285    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:17:43.723439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.723405    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:43.723736 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.723708    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:43.736079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.736029    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podStartSLOduration=11.736015674 podStartE2EDuration="11.736015674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:41.224521337 +0000 UTC m=+848.866427669" watchObservedRunningTime="2026-11-04 01:17:43.736015674 +0000 UTC m=+851.377921921"
Nov 04 01:17:43.746039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.745962    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"]
Nov 04 01:17:43.816221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.816166    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.819573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.819414    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"]
Nov 04 01:17:43.827790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.827638    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"]
Nov 04 01:17:43.838331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.838307    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:43.839939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.839904    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.840054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.840037    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.840130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.840096    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkmd\" (UniqueName: \"kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.840277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.840261    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"]
Nov 04 01:17:43.926292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.926253    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2" event={"ID":"178c4ef8-17dd-41bf-b73e-66a53ef646c0","Type":"ContainerStarted","Data":"54bf91dac1c88402ce0f3300ff12b760b50f64319d6a1a6d8e8acc7d351b0093"}
Nov 04 01:17:43.939567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.939541    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"]
Nov 04 01:17:43.942153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.942118    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wmx\" (UniqueName: \"kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:43.942272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.942168    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.942272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.942200    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:43.943549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.943525    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.943638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.943571    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkmd\" (UniqueName: \"kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.943638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.943603    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:43.945223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.945201    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:43.946541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:43.946516    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:44.005054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.005009    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:44.007049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.007009    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkmd\" (UniqueName: \"kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd\") pod \"maas-api-revoke-sub-69e4eefc-hhvw5\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:44.115024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.106127    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.128960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.128871    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wmx\" (UniqueName: \"kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.129240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.129219    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.132982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.132940    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.145711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.145677    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.280573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.272151    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:44.280573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.272215    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:44.313023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.312640    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:44.341516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.340190    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wmx\" (UniqueName: \"kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx\") pod \"maas-api-revoke-sub-b3af645a-h4svc\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.542431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.541279    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:44.710075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.702256    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:44.710075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.702315    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:44.939621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:44.939510    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2" event={"ID":"178c4ef8-17dd-41bf-b73e-66a53ef646c0","Type":"ContainerStarted","Data":"7dc8f1616641d62ec5c5c084cfcdf0c7ce148c0b863cedf9139ea6bbc6688a57"}
Nov 04 01:17:45.014911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.014796    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2" podStartSLOduration=7.014775103 podStartE2EDuration="7.014775103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:44.98767239 +0000 UTC m=+852.629578667" watchObservedRunningTime="2026-11-04 01:17:45.014775103 +0000 UTC m=+852.656681354"
Nov 04 01:17:45.031736 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.031708    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"]
Nov 04 01:17:45.214238 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.214202    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:45.214483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.214454    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:45.223533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.223507    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"]
Nov 04 01:17:45.392043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.391937    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:45.689017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.688691    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:45.689017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.688751    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:45.976704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.976662    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc" event={"ID":"77353796-050c-449b-9614-a778f6c68f3e","Type":"ContainerStarted","Data":"e1ed90c25cac5000b8ebf2194557e19fc5d4cc487959234edec4480741fac22c"}
Nov 04 01:17:45.991958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.991922    5797 generic.go:356] "Generic (PLEG): container finished" podID="178c4ef8-17dd-41bf-b73e-66a53ef646c0" containerID="7dc8f1616641d62ec5c5c084cfcdf0c7ce148c0b863cedf9139ea6bbc6688a57" exitCode=0
Nov 04 01:17:45.997240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:45.997203    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2" event={"ID":"178c4ef8-17dd-41bf-b73e-66a53ef646c0","Type":"ContainerDied","Data":"7dc8f1616641d62ec5c5c084cfcdf0c7ce148c0b863cedf9139ea6bbc6688a57"}
Nov 04 01:17:46.016026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.014032    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5" event={"ID":"79fa7fcc-738c-4e55-b73a-859b679f501a","Type":"ContainerStarted","Data":"fdebb6c6a69c931c2a95c8b5ab68b99f4620eccc207fe83f00a5c8d3d5601679"}
Nov 04 01:17:46.016026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.014111    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5" event={"ID":"79fa7fcc-738c-4e55-b73a-859b679f501a","Type":"ContainerStarted","Data":"3d23fc2d4a00b9d34116d7f9a5cbc158619ccc8e7df80f767d015f0e6736ff54"}
Nov 04 01:17:46.020057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.018305    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"]
Nov 04 01:17:46.026094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.025849    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"]
Nov 04 01:17:46.026094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.025952    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.039951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.039063    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.039951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.039107    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6x5p\" (UniqueName: \"kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.039951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.039136    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.163015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.162376    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.163015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.162428    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6x5p\" (UniqueName: \"kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.163015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.162461    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.195165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.165370    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5" podStartSLOduration=5.165350773 podStartE2EDuration="5.165350773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:46.04638313 +0000 UTC m=+853.688289380" watchObservedRunningTime="2026-11-04 01:17:46.165350773 +0000 UTC m=+853.807257024"
Nov 04 01:17:46.314073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.214183    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.314073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.219291    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.314073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.308942    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:46.314073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.312417    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:46.327389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.317311    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6x5p\" (UniqueName: \"kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p\") pod \"maas-api-revoke-sub-e75ddfb3-zcj99\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.639693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.639664    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:46.645780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.645753    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:46.686487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.686449    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:46.688513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:46.688476    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:47.029454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.029365    5797 generic.go:356] "Generic (PLEG): container finished" podID="77353796-050c-449b-9614-a778f6c68f3e" containerID="92688b29d5149f8d4073f90889015340f25db57ff2f31c42540358bcfac6b71b" exitCode=0
Nov 04 01:17:47.029906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.029464    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc" event={"ID":"77353796-050c-449b-9614-a778f6c68f3e","Type":"ContainerDied","Data":"92688b29d5149f8d4073f90889015340f25db57ff2f31c42540358bcfac6b71b"}
Nov 04 01:17:47.031285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.031261    5797 generic.go:356] "Generic (PLEG): container finished" podID="79fa7fcc-738c-4e55-b73a-859b679f501a" containerID="fdebb6c6a69c931c2a95c8b5ab68b99f4620eccc207fe83f00a5c8d3d5601679" exitCode=0
Nov 04 01:17:47.031434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.031330    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5" event={"ID":"79fa7fcc-738c-4e55-b73a-859b679f501a","Type":"ContainerDied","Data":"fdebb6c6a69c931c2a95c8b5ab68b99f4620eccc207fe83f00a5c8d3d5601679"}
Nov 04 01:17:47.131675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.131642    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"]
Nov 04 01:17:47.214549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.214511    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:47.214730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.214575    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:47.748025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.742335    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:47.748025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.742398    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:47.774021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.771557    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:47.823279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.822723    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:47.967700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.967658    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca\") pod \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") "
Nov 04 01:17:47.969382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.969357    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx\" (UniqueName: \"kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx\") pod \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") "
Nov 04 01:17:47.969974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.969952    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca\") pod \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\" (UID: \"178c4ef8-17dd-41bf-b73e-66a53ef646c0\") "
Nov 04 01:17:47.981535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:47.981475    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca" pod "178c4ef8-17dd-41bf-b73e-66a53ef646c0" (UID: "178c4ef8-17dd-41bf-b73e-66a53ef646c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:48.002050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.001019    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.008628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.008576    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx" pod "178c4ef8-17dd-41bf-b73e-66a53ef646c0" (UID: "178c4ef8-17dd-41bf-b73e-66a53ef646c0"). InnerVolumeSpecName "kube-api-access-h46jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:48.015985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.015943    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca" pod "178c4ef8-17dd-41bf-b73e-66a53ef646c0" (UID: "178c4ef8-17dd-41bf-b73e-66a53ef646c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:48.055403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.052850    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99" event={"ID":"2f17b85f-4d63-4903-9a1c-9bd4c516be8c","Type":"ContainerStarted","Data":"bc15ec25e9ad4e7ce6e7028021d06f405183426582933dfc3b3803294b6289ef"}
Nov 04 01:17:48.055403 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.052918    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99" event={"ID":"2f17b85f-4d63-4903-9a1c-9bd4c516be8c","Type":"ContainerStarted","Data":"f560e7d756ce81dd452ffcd9cb297b4ba6dbfda14c12d08f81d0759ccd2b4a76"}
Nov 04 01:17:48.081635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.081551    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"
Nov 04 01:17:48.090102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.090051    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2" event={"ID":"178c4ef8-17dd-41bf-b73e-66a53ef646c0","Type":"ContainerDied","Data":"54bf91dac1c88402ce0f3300ff12b760b50f64319d6a1a6d8e8acc7d351b0093"}
Nov 04 01:17:48.090295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.090126    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54bf91dac1c88402ce0f3300ff12b760b50f64319d6a1a6d8e8acc7d351b0093"
Nov 04 01:17:48.102881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.102457    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c4ef8-17dd-41bf-b73e-66a53ef646c0-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.102881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.102480    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46jx\" (UniqueName: \"kubernetes.io/projected/178c4ef8-17dd-41bf-b73e-66a53ef646c0-kube-api-access-h46jx\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.109942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.109884    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99" podStartSLOduration=3.109863604 podStartE2EDuration="3.109863604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:48.107575983 +0000 UTC m=+855.749482248" watchObservedRunningTime="2026-11-04 01:17:48.109863604 +0000 UTC m=+855.751769907"
Nov 04 01:17:48.213490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.213084    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:48.213490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.213165    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:48.235965 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.235889    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:48.439465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.439427    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"]
Nov 04 01:17:48.441257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.441238    5797 cpu_manager.go:414] "containerMap: removing container" podUID="178c4ef8-17dd-41bf-b73e-66a53ef646c0" containerName="revoke-keys"
Nov 04 01:17:48.441380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.441371    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178c4ef8-17dd-41bf-b73e-66a53ef646c0" containerName="revoke-keys"
Nov 04 01:17:48.441594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.441584    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="178c4ef8-17dd-41bf-b73e-66a53ef646c0" containerName="revoke-keys"
Nov 04 01:17:48.447498 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.447477    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.532021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.530324    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbtl\" (UniqueName: \"kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.532021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.530447    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.532021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.530702    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.532021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.530971    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:48.541947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.541922    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:48.613195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.613120    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"]
Nov 04 01:17:48.632400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.632360    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca\") pod \"77353796-050c-449b-9614-a778f6c68f3e\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") "
Nov 04 01:17:48.632672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.632651    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx\" (UniqueName: \"kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx\") pod \"77353796-050c-449b-9614-a778f6c68f3e\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") "
Nov 04 01:17:48.632843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.632828    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca\") pod \"79fa7fcc-738c-4e55-b73a-859b679f501a\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") "
Nov 04 01:17:48.633033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.633017    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd\" (UniqueName: \"kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd\") pod \"79fa7fcc-738c-4e55-b73a-859b679f501a\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") "
Nov 04 01:17:48.633299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.633280    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca\") pod \"79fa7fcc-738c-4e55-b73a-859b679f501a\" (UID: \"79fa7fcc-738c-4e55-b73a-859b679f501a\") "
Nov 04 01:17:48.633520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.633501    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca\") pod \"77353796-050c-449b-9614-a778f6c68f3e\" (UID: \"77353796-050c-449b-9614-a778f6c68f3e\") "
Nov 04 01:17:48.633865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.633842    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.634222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.634201    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.634503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.634485    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbtl\" (UniqueName: \"kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.635431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.635406    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca" pod "77353796-050c-449b-9614-a778f6c68f3e" (UID: "77353796-050c-449b-9614-a778f6c68f3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:48.636620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.636592    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca" pod "79fa7fcc-738c-4e55-b73a-859b679f501a" (UID: "79fa7fcc-738c-4e55-b73a-859b679f501a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:48.637797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.637774    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.644139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.638517    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx" pod "77353796-050c-449b-9614-a778f6c68f3e" (UID: "77353796-050c-449b-9614-a778f6c68f3e"). InnerVolumeSpecName "kube-api-access-l9wmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:48.644139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.640789    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.644302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.644076    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca" pod "79fa7fcc-738c-4e55-b73a-859b679f501a" (UID: "79fa7fcc-738c-4e55-b73a-859b679f501a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:48.644617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.644595    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd" pod "79fa7fcc-738c-4e55-b73a-859b679f501a" (UID: "79fa7fcc-738c-4e55-b73a-859b679f501a"). InnerVolumeSpecName "kube-api-access-hzkmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:48.646691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.646648    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca" pod "77353796-050c-449b-9614-a778f6c68f3e" (UID: "77353796-050c-449b-9614-a778f6c68f3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:48.680057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.658640    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbtl\" (UniqueName: \"kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl\") pod \"maas-api-revoke-sub-bca75a03-tl64z\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:48.687644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.687605    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:48.688787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.688759    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:48.688946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.688800    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:48.737679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.736939    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.737679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.736979    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77353796-050c-449b-9614-a778f6c68f3e-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.737679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.737011    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77353796-050c-449b-9614-a778f6c68f3e-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.737679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.737027    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9wmx\" (UniqueName: \"kubernetes.io/projected/77353796-050c-449b-9614-a778f6c68f3e-kube-api-access-l9wmx\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.737679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.737040    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fa7fcc-738c-4e55-b73a-859b679f501a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.737679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.737053    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkmd\" (UniqueName: \"kubernetes.io/projected/79fa7fcc-738c-4e55-b73a-859b679f501a-kube-api-access-hzkmd\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:48.872887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.872706    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:48.899129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:48.899094    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:49.110705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.110677    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"
Nov 04 01:17:49.111462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.110782    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc" event={"ID":"77353796-050c-449b-9614-a778f6c68f3e","Type":"ContainerDied","Data":"e1ed90c25cac5000b8ebf2194557e19fc5d4cc487959234edec4480741fac22c"}
Nov 04 01:17:49.111594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.111531    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ed90c25cac5000b8ebf2194557e19fc5d4cc487959234edec4480741fac22c"
Nov 04 01:17:49.113175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.113144    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5" event={"ID":"79fa7fcc-738c-4e55-b73a-859b679f501a","Type":"ContainerDied","Data":"3d23fc2d4a00b9d34116d7f9a5cbc158619ccc8e7df80f767d015f0e6736ff54"}
Nov 04 01:17:49.113271 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.113197    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d23fc2d4a00b9d34116d7f9a5cbc158619ccc8e7df80f767d015f0e6736ff54"
Nov 04 01:17:49.113347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.113271    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"
Nov 04 01:17:49.125051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.123487    5797 generic.go:356] "Generic (PLEG): container finished" podID="2f17b85f-4d63-4903-9a1c-9bd4c516be8c" containerID="bc15ec25e9ad4e7ce6e7028021d06f405183426582933dfc3b3803294b6289ef" exitCode=0
Nov 04 01:17:49.125051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.123562    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99" event={"ID":"2f17b85f-4d63-4903-9a1c-9bd4c516be8c","Type":"ContainerDied","Data":"bc15ec25e9ad4e7ce6e7028021d06f405183426582933dfc3b3803294b6289ef"}
Nov 04 01:17:49.213599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.213564    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:17:49.213825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.213800    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:17:49.241698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.241675    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"]
Nov 04 01:17:49.244136 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:49.244103    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df782c7_7aab_4768_935d_944a5bb83068.slice/crio-2e9a0bfa852ec989218a8af44f43f83c6f10401337531b2910f6a6a5204a54df WatchSource:0}: Error finding container 2e9a0bfa852ec989218a8af44f43f83c6f10401337531b2910f6a6a5204a54df: Status 404 returned error can't find the container with id 2e9a0bfa852ec989218a8af44f43f83c6f10401337531b2910f6a6a5204a54df
Nov 04 01:17:49.687161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.687130    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:17:49.687320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.687184    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:17:49.791841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:49.791184    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:50.137922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.137884    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z" event={"ID":"1df782c7-7aab-4768-935d-944a5bb83068","Type":"ContainerStarted","Data":"e405010b2eb2a72aaf3d0daa267df5d63d2fcd5f99c9d73d76619003a642ee27"}
Nov 04 01:17:50.138457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.137947    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z" event={"ID":"1df782c7-7aab-4768-935d-944a5bb83068","Type":"ContainerStarted","Data":"2e9a0bfa852ec989218a8af44f43f83c6f10401337531b2910f6a6a5204a54df"}
Nov 04 01:17:50.161772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.161697    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z" podStartSLOduration=2.161677005 podStartE2EDuration="2.161677005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:50.160309524 +0000 UTC m=+857.802215809" watchObservedRunningTime="2026-11-04 01:17:50.161677005 +0000 UTC m=+857.803583256"
Nov 04 01:17:50.234434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.231443    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:50.234434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.234229    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:17:50.443633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.443553    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:50.504050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.504017    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p\" (UniqueName: \"kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p\") pod \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") "
Nov 04 01:17:50.504246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.504097    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca\") pod \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") "
Nov 04 01:17:50.504358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.504337    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca\") pod \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\" (UID: \"2f17b85f-4d63-4903-9a1c-9bd4c516be8c\") "
Nov 04 01:17:50.505369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.505334    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca" pod "2f17b85f-4d63-4903-9a1c-9bd4c516be8c" (UID: "2f17b85f-4d63-4903-9a1c-9bd4c516be8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:50.507151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.507128    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p" pod "2f17b85f-4d63-4903-9a1c-9bd4c516be8c" (UID: "2f17b85f-4d63-4903-9a1c-9bd4c516be8c"). InnerVolumeSpecName "kube-api-access-g6x5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:50.507568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.507533    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca" pod "2f17b85f-4d63-4903-9a1c-9bd4c516be8c" (UID: "2f17b85f-4d63-4903-9a1c-9bd4c516be8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:50.614045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.610702    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6x5p\" (UniqueName: \"kubernetes.io/projected/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-kube-api-access-g6x5p\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:50.614045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.610746    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:50.614045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.610770    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f17b85f-4d63-4903-9a1c-9bd4c516be8c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:50.649850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.649812    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:50.794646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.794562    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:50.795740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:50.795718    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:17:51.149318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:51.149284    5797 generic.go:356] "Generic (PLEG): container finished" podID="1df782c7-7aab-4768-935d-944a5bb83068" containerID="e405010b2eb2a72aaf3d0daa267df5d63d2fcd5f99c9d73d76619003a642ee27" exitCode=0
Nov 04 01:17:51.149810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:51.149371    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z" event={"ID":"1df782c7-7aab-4768-935d-944a5bb83068","Type":"ContainerDied","Data":"e405010b2eb2a72aaf3d0daa267df5d63d2fcd5f99c9d73d76619003a642ee27"}
Nov 04 01:17:51.151088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:51.151069    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"
Nov 04 01:17:51.151227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:51.151135    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99" event={"ID":"2f17b85f-4d63-4903-9a1c-9bd4c516be8c","Type":"ContainerDied","Data":"f560e7d756ce81dd452ffcd9cb297b4ba6dbfda14c12d08f81d0759ccd2b4a76"}
Nov 04 01:17:51.151227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:51.151196    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f560e7d756ce81dd452ffcd9cb297b4ba6dbfda14c12d08f81d0759ccd2b4a76"
Nov 04 01:17:51.409233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:51.409132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:52.305672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.305202    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:52.438927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.438900    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:52.543350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.543307    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl\" (UniqueName: \"kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl\") pod \"1df782c7-7aab-4768-935d-944a5bb83068\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") "
Nov 04 01:17:52.543520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.543451    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca\") pod \"1df782c7-7aab-4768-935d-944a5bb83068\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") "
Nov 04 01:17:52.543655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.543636    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca\") pod \"1df782c7-7aab-4768-935d-944a5bb83068\" (UID: \"1df782c7-7aab-4768-935d-944a5bb83068\") "
Nov 04 01:17:52.544059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.544031    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca" pod "1df782c7-7aab-4768-935d-944a5bb83068" (UID: "1df782c7-7aab-4768-935d-944a5bb83068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:17:52.544702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.544671    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:52.545723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.545696    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca" pod "1df782c7-7aab-4768-935d-944a5bb83068" (UID: "1df782c7-7aab-4768-935d-944a5bb83068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:17:52.545801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.545773    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl" pod "1df782c7-7aab-4768-935d-944a5bb83068" (UID: "1df782c7-7aab-4768-935d-944a5bb83068"). InnerVolumeSpecName "kube-api-access-nhbtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:17:52.646311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.646252    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhbtl\" (UniqueName: \"kubernetes.io/projected/1df782c7-7aab-4768-935d-944a5bb83068-kube-api-access-nhbtl\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:52.646311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:52.646288    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df782c7-7aab-4768-935d-944a5bb83068-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:17:53.172261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:53.172228    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"
Nov 04 01:17:53.172448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:53.172268    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z" event={"ID":"1df782c7-7aab-4768-935d-944a5bb83068","Type":"ContainerDied","Data":"2e9a0bfa852ec989218a8af44f43f83c6f10401337531b2910f6a6a5204a54df"}
Nov 04 01:17:53.172448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:53.172327    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9a0bfa852ec989218a8af44f43f83c6f10401337531b2910f6a6a5204a54df"
Nov 04 01:17:53.202390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:53.202357    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:54.004673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.004636    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"]
Nov 04 01:17:54.005339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005325    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2f17b85f-4d63-4903-9a1c-9bd4c516be8c" containerName="revoke-keys"
Nov 04 01:17:54.005384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005340    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f17b85f-4d63-4903-9a1c-9bd4c516be8c" containerName="revoke-keys"
Nov 04 01:17:54.005384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005352    5797 cpu_manager.go:414] "containerMap: removing container" podUID="79fa7fcc-738c-4e55-b73a-859b679f501a" containerName="revoke-keys"
Nov 04 01:17:54.005384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005358    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fa7fcc-738c-4e55-b73a-859b679f501a" containerName="revoke-keys"
Nov 04 01:17:54.005489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005394    5797 cpu_manager.go:414] "containerMap: removing container" podUID="77353796-050c-449b-9614-a778f6c68f3e" containerName="revoke-keys"
Nov 04 01:17:54.005489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005400    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77353796-050c-449b-9614-a778f6c68f3e" containerName="revoke-keys"
Nov 04 01:17:54.005489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005407    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1df782c7-7aab-4768-935d-944a5bb83068" containerName="revoke-keys"
Nov 04 01:17:54.005489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005411    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df782c7-7aab-4768-935d-944a5bb83068" containerName="revoke-keys"
Nov 04 01:17:54.005603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005518    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f17b85f-4d63-4903-9a1c-9bd4c516be8c" containerName="revoke-keys"
Nov 04 01:17:54.005603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005529    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fa7fcc-738c-4e55-b73a-859b679f501a" containerName="revoke-keys"
Nov 04 01:17:54.005603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005539    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="77353796-050c-449b-9614-a778f6c68f3e" containerName="revoke-keys"
Nov 04 01:17:54.005603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.005544    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df782c7-7aab-4768-935d-944a5bb83068" containerName="revoke-keys"
Nov 04 01:17:54.022438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.022399    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.027946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.027308    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:17:54.027946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.027803    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:17:54.030334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.030310    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"]
Nov 04 01:17:54.172891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.171592    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.172891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.171683    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbfh\" (UniqueName: \"kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.172891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.171766    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.199321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.199287    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:54.273762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.273679    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.273762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.273736    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbfh\" (UniqueName: \"kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.273945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.273775    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.274524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.274498    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.276305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.276279    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.286641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.286616    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbfh\" (UniqueName: \"kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh\") pod \"maas-api-revoke-sub-5e6b8db9-vjlrg\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.344633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.344605    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:17:54.633443 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:17:54.633412    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70346d7c_b077_481d_a38a_64489f7caea3.slice/crio-b7792d1017c514121c0012ec66cc2800686381059f241b920415f98b21e8c116 WatchSource:0}: Error finding container b7792d1017c514121c0012ec66cc2800686381059f241b920415f98b21e8c116: Status 404 returned error can't find the container with id b7792d1017c514121c0012ec66cc2800686381059f241b920415f98b21e8c116
Nov 04 01:17:54.636582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:54.636554    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"]
Nov 04 01:17:55.046499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:55.046463    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:55.213376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:55.213275    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg" event={"ID":"70346d7c-b077-481d-a38a-64489f7caea3","Type":"ContainerStarted","Data":"86151189bae7c0be2abf0d0bccb8ef3ff2bedc814f4271d03bcfef4d375637dd"}
Nov 04 01:17:55.213376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:55.213334    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg" event={"ID":"70346d7c-b077-481d-a38a-64489f7caea3","Type":"ContainerStarted","Data":"b7792d1017c514121c0012ec66cc2800686381059f241b920415f98b21e8c116"}
Nov 04 01:17:55.236254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:55.236207    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg" podStartSLOduration=2.236193696 podStartE2EDuration="2.236193696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:55.231299089 +0000 UTC m=+862.873205372" watchObservedRunningTime="2026-11-04 01:17:55.236193696 +0000 UTC m=+862.878099946"
Nov 04 01:17:55.811446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:55.811413    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:56.313043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:56.307660    5797 generic.go:356] "Generic (PLEG): container finished" podID="70346d7c-b077-481d-a38a-64489f7caea3" containerID="86151189bae7c0be2abf0d0bccb8ef3ff2bedc814f4271d03bcfef4d375637dd" exitCode=0
Nov 04 01:17:56.313043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:56.307966    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg" event={"ID":"70346d7c-b077-481d-a38a-64489f7caea3","Type":"ContainerDied","Data":"86151189bae7c0be2abf0d0bccb8ef3ff2bedc814f4271d03bcfef4d375637dd"}
Nov 04 01:17:57.525195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:57.525128    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:57.730065 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:17:57.721033    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:17:57.730065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:57.723441    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:57.959014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:57.949113    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:57.959014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:57.949567    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:17:58.147019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.135332    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:58.306242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.275560    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:58.306242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.280475    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:58.450065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.424258    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:58.537639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.452417    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:58.573249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.498105    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:58.577532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.577055    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:17:58.742064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.735656    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:17:58.829353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.829314    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:58.990044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:58.976102    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:59.144014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.125940    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"]
Nov 04 01:17:59.744162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.742822    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.676082    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.677308    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.689740    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.690702    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.690733    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.697429    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.698489    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.747863    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:17:59.762595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.534585    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:00.009260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:17:59.994757    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:00.091560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.091532    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"]
Nov 04 01:18:00.091839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.091821    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:18:00.097104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.096962    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.097254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.097237    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:18:00.101027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.097744    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:18:00.168119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.168089    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:18:00.178201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.178165    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:18:00.178489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.178471    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:18:00.204016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.178622    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:18:00.204469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.204440    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:18:00.205560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.205528    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcql\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.205928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.205904    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.206219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.206198    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.206421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.206404    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.225029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.129808    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:00.255011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.206612    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.255389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.255366    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.265221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.265133    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.265501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.265480    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.276158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.276127    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.313931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.129181    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9cc22d46-openshift-default-dockercfg-2ghmg\""
Nov 04 01:18:00.364015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.353099    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:18:00.470016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.444692    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca\") pod \"70346d7c-b077-481d-a38a-64489f7caea3\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") "
Nov 04 01:18:00.470938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.470898    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\") pod \"70346d7c-b077-481d-a38a-64489f7caea3\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") "
Nov 04 01:18:00.471206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.471180    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh\" (UniqueName: \"kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh\") pod \"70346d7c-b077-481d-a38a-64489f7caea3\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") "
Nov 04 01:18:00.471657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.471630    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.471793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.471775    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.471959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.471944    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcql\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.476011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.472358    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.482957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.482929    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.490018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.483274    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.490417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.490391    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.497177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.497148    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.523070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.497411    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.558060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.523654    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.558060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.523869    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.558060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.476560    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca" pod "70346d7c-b077-481d-a38a-64489f7caea3" (UID: "70346d7c-b077-481d-a38a-64489f7caea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:00.617221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.582190    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.617221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.583185    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.680117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.680070    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.685263    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.685323    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.686171    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.686226    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.689020    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.726611    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.767148    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.767532    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.767810    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.768073    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.768300    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:18:00.786075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.768520    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:00.812084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.796875    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca" pod "70346d7c-b077-481d-a38a-64489f7caea3" (UID: "70346d7c-b077-481d-a38a-64489f7caea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:00.861014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.816892    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:00.861014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.818362    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:00.861014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.840661    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\") pod \"70346d7c-b077-481d-a38a-64489f7caea3\" (UID: \"70346d7c-b077-481d-a38a-64489f7caea3\") "
Nov 04 01:18:00.891021 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:00.875206    5797 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/70346d7c-b077-481d-a38a-64489f7caea3/volumes/kubernetes.io~secret/maas-api-xks-ca
Nov 04 01:18:00.891021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.875236    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca" pod "70346d7c-b077-481d-a38a-64489f7caea3" (UID: "70346d7c-b077-481d-a38a-64489f7caea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:00.891021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.880152    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:00.891021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:00.880184    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70346d7c-b077-481d-a38a-64489f7caea3-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:01.347018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.345696    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:01.383014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.373953    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:01.383014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.382251    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:01.490231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.490187    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcql\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql\") pod \"e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") " pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:01.561016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.547400    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh" pod "70346d7c-b077-481d-a38a-64489f7caea3" (UID: "70346d7c-b077-481d-a38a-64489f7caea3"). InnerVolumeSpecName "kube-api-access-lqbfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:01.915336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.915282    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Nov 04 01:18:01.926018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.922216    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:01.977015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.951521    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:02.041016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.959098    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:02.041016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.959138    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:02.041016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.959172    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:02.041016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:01.959196    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:18:02.041016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:02.003201    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqbfh\" (UniqueName: \"kubernetes.io/projected/70346d7c-b077-481d-a38a-64489f7caea3-kube-api-access-lqbfh\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:02.112092 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:02.049208    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.322s"
Nov 04 01:18:02.112092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:02.110620    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:02.160933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:02.160896    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:02.392139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:02.351243    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:02.394976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:02.394934    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded"
Nov 04 01:18:02.966048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:02.955499    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:03.083167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.077016    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:03.096519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.095481    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:03.096519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.095730    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:03.096519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.095875    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:03.096519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.096037    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:03.096519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.096168    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:03.118045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.115291    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"
Nov 04 01:18:03.141015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.122103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:03.195014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.178572    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 04 01:18:03.244064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.222008    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg" event={"ID":"70346d7c-b077-481d-a38a-64489f7caea3","Type":"ContainerDied","Data":"b7792d1017c514121c0012ec66cc2800686381059f241b920415f98b21e8c116"}
Nov 04 01:18:03.244064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.222072    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7792d1017c514121c0012ec66cc2800686381059f241b920415f98b21e8c116"
Nov 04 01:18:03.361017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:03.354578    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:04.176624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.176534    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 04 01:18:04.177422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.177392    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.177584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.177558    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.184294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.184261    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.184650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.184624    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:04.184933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.177507    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.199384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.177537    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.200289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.199806    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.388099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.387502    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:04.537023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.536896    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:04.703219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.703173    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:04.703469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.703446    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:04.703638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.703617    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:04.786635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.786597    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:04.879302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.878586    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"]
Nov 04 01:18:04.992009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.988473    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:18:04.999068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:04.999043    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:18:05.048073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.048040    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:18:05.135385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.135355    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:18:05.151865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.151737    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:18:05.155814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.153276    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:18:05.196076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.195381    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:05.199507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.199425    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:05.223650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.223441    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:18:05.230173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.230149    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:18:05.230910 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.230875    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" event={"ID":"0039b909-973d-4845-bb8d-db4dd4c57445","Type":"ContainerStarted","Data":"769dabfd31dc0074fbebdfe455edf06c2ad658ff8b98143b95ce503bb81bcd2e"}
Nov 04 01:18:05.308001 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:05.307950    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:06.115905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:06.115874    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:06.249668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:06.249623    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" event={"ID":"0039b909-973d-4845-bb8d-db4dd4c57445","Type":"ContainerStarted","Data":"406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3"}
Nov 04 01:18:06.280533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:06.280478    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" podStartSLOduration=8.280461598 podStartE2EDuration="8.280461598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:06.275271645 +0000 UTC m=+873.917177895" watchObservedRunningTime="2026-11-04 01:18:06.280461598 +0000 UTC m=+873.922367943"
Nov 04 01:18:07.119709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.119669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:07.187926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.187889    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:07.192981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.192954    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:07.262912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.262881    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:07.263798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.263770    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:07.675670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.675639    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:07.958062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:07.954568    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:08.106908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.106816    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"]
Nov 04 01:18:08.110423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.108811    5797 cpu_manager.go:414] "containerMap: removing container" podUID="70346d7c-b077-481d-a38a-64489f7caea3" containerName="revoke-keys"
Nov 04 01:18:08.110423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.108831    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70346d7c-b077-481d-a38a-64489f7caea3" containerName="revoke-keys"
Nov 04 01:18:08.110423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.109035    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="70346d7c-b077-481d-a38a-64489f7caea3" containerName="revoke-keys"
Nov 04 01:18:08.115682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.114874    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.136011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.134507    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-383a864b-openshift-default-dockercfg-kng4j\""
Nov 04 01:18:08.138890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.138141    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"]
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.162736    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.162789    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n882d\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.162944    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.163111    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.163173    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.163222    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.163276    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.163364    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.163840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.163409    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.223301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.223216    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:08.264854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.264819    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.264896    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265118    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265153    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n882d\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265194    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265293    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265341    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265388    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265416    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265444    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265719    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.265805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265796    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.266020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.265962    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.266205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.266181    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.268355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.268328    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.281199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.281167    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.286804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.286779    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.287272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.287245    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n882d\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d\") pod \"e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") " pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.460631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.460160    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:08.494444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.494416    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:08.746895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.746459    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:08.806376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.805207    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"]
Nov 04 01:18:08.808959 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:08.808928    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fb7b17_2b56_486a_8c75_051f82dd7045.slice/crio-68337384be52d7134414cad2a706c632d0e282c49b3341b7dab12712e58cecaa WatchSource:0}: Error finding container 68337384be52d7134414cad2a706c632d0e282c49b3341b7dab12712e58cecaa: Status 404 returned error can't find the container with id 68337384be52d7134414cad2a706c632d0e282c49b3341b7dab12712e58cecaa
Nov 04 01:18:08.812446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.812396    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:08.812576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.812475    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:08.812576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:08.812517    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:09.293124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.293089    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" event={"ID":"69fb7b17-2b56-486a-8c75-051f82dd7045","Type":"ContainerStarted","Data":"20f029a846e7c28edd1260f7e501b5487b8dfb4fa05ec73860924c79433c310e"}
Nov 04 01:18:09.293594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.293150    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" event={"ID":"69fb7b17-2b56-486a-8c75-051f82dd7045","Type":"ContainerStarted","Data":"68337384be52d7134414cad2a706c632d0e282c49b3341b7dab12712e58cecaa"}
Nov 04 01:18:09.321082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.321027    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" podStartSLOduration=1.321009081 podStartE2EDuration="1.321009081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:09.317822909 +0000 UTC m=+876.959729183" watchObservedRunningTime="2026-11-04 01:18:09.321009081 +0000 UTC m=+876.962915333"
Nov 04 01:18:09.378680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.378641    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:09.504013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.479291    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:09.534835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.534801    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:09.596814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:09.596391    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:10.305037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:10.304997    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:10.306034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:10.305979    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:18:10.702319 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:10.700761    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:11.085197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:11.085085    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:11.300269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:11.300232    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:11.878282 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:11.877950    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:12.320576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:12.320487    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:12.605484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:12.605351    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:12.906524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:12.906484    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:13.294486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:13.293677    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:13.516947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:13.514006    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:13.775159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:13.775107    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:14.003013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:14.000106    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:14.203867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:14.203827    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:14.697645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:14.697614    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:14.913475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:14.913436    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:15.122033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:15.121953    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:15.901011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:15.894452    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:16.125817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:16.125782    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:16.322021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:16.321920    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:16.953012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:16.947678    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:17.983137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:17.982336    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:18.245077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.241036    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"]
Nov 04 01:18:18.255302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.255269    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"]
Nov 04 01:18:18.255525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.255393    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.261720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.261689    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:18:18.264047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.262211    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:18:18.383138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.383108    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whhn\" (UniqueName: \"kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.383341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.383246    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.383510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.383495    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.484863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.484831    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.485060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.484905    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whhn\" (UniqueName: \"kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.485060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.484958    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.485633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.485609    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.487269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.487247    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.497920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.497872    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whhn\" (UniqueName: \"kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn\") pod \"maas-api-revoke-sub-f8ea69b7-qw598\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.578166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.578136    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:18.868362 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:18.868332    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9521b723_3f93_4e36_8c9c_c415cf6f82e1.slice/crio-54fa76960f2a3254701f210c18c1cde7b7b9cf992666064f9e55d219af5a0ac8 WatchSource:0}: Error finding container 54fa76960f2a3254701f210c18c1cde7b7b9cf992666064f9e55d219af5a0ac8: Status 404 returned error can't find the container with id 54fa76960f2a3254701f210c18c1cde7b7b9cf992666064f9e55d219af5a0ac8
Nov 04 01:18:18.871437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:18.871413    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"]
Nov 04 01:18:19.029718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.029657    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:19.481448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.481415    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"]
Nov 04 01:18:19.486212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.486186    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.496544 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.496502    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"]
Nov 04 01:18:19.606323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.606291    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.606482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.606362    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.606482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.606474    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26z2\" (UniqueName: \"kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.622672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.622643    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598" event={"ID":"9521b723-3f93-4e36-8c9c-c415cf6f82e1","Type":"ContainerStarted","Data":"e7f72026cbfa956687e8d4ae8eaa5ef35e8d5344d8fd73009ac60698af8b8fbc"}
Nov 04 01:18:19.622806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.622692    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598" event={"ID":"9521b723-3f93-4e36-8c9c-c415cf6f82e1","Type":"ContainerStarted","Data":"54fa76960f2a3254701f210c18c1cde7b7b9cf992666064f9e55d219af5a0ac8"}
Nov 04 01:18:19.643695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.643653    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598" podStartSLOduration=1.643639226 podStartE2EDuration="1.643639226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:19.64026101 +0000 UTC m=+887.282167283" watchObservedRunningTime="2026-11-04 01:18:19.643639226 +0000 UTC m=+887.285545476"
Nov 04 01:18:19.708467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.708416    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.708646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.708508    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.708646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.708604    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26z2\" (UniqueName: \"kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.709573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.709542    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.711601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.711573    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.726134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.725982    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26z2\" (UniqueName: \"kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2\") pod \"maas-api-revoke-sub-55be8faa-cz7gf\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.824817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.824743    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:19.852159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:19.852119    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:20.053835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.053790    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:20.117661 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:20.117625    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85280e39_4b0c_4515_8889_e81e6e77ee3b.slice/crio-eaf8c10a75fb4663e2164aef40ad32ea3693a41f446666b8a4cf1ae67d41cf88 WatchSource:0}: Error finding container eaf8c10a75fb4663e2164aef40ad32ea3693a41f446666b8a4cf1ae67d41cf88: Status 404 returned error can't find the container with id eaf8c10a75fb4663e2164aef40ad32ea3693a41f446666b8a4cf1ae67d41cf88
Nov 04 01:18:20.126012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.122229    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"]
Nov 04 01:18:20.645841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.645756    5797 generic.go:356] "Generic (PLEG): container finished" podID="9521b723-3f93-4e36-8c9c-c415cf6f82e1" containerID="e7f72026cbfa956687e8d4ae8eaa5ef35e8d5344d8fd73009ac60698af8b8fbc" exitCode=0
Nov 04 01:18:20.646019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.645833    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598" event={"ID":"9521b723-3f93-4e36-8c9c-c415cf6f82e1","Type":"ContainerDied","Data":"e7f72026cbfa956687e8d4ae8eaa5ef35e8d5344d8fd73009ac60698af8b8fbc"}
Nov 04 01:18:20.651059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.651022    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf" event={"ID":"85280e39-4b0c-4515-8889-e81e6e77ee3b","Type":"ContainerStarted","Data":"6287560de98dc57e73ae890b17ab42319fa25ffb0920ab338742aa3075943983"}
Nov 04 01:18:20.651249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.651087    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf" event={"ID":"85280e39-4b0c-4515-8889-e81e6e77ee3b","Type":"ContainerStarted","Data":"eaf8c10a75fb4663e2164aef40ad32ea3693a41f446666b8a4cf1ae67d41cf88"}
Nov 04 01:18:20.689478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.689418    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf" podStartSLOduration=1.689366095 podStartE2EDuration="1.689366095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:20.683067219 +0000 UTC m=+888.324973493" watchObservedRunningTime="2026-11-04 01:18:20.689366095 +0000 UTC m=+888.331272346"
Nov 04 01:18:20.822094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.816976    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"]
Nov 04 01:18:20.825428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.824922    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:20.829328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.829302    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"]
Nov 04 01:18:20.967397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.957814    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:20.967397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.967215    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:20.967639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.967449    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lslnf\" (UniqueName: \"kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:20.967639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:20.967579    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.069784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.069749    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.070230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.069902    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lslnf\" (UniqueName: \"kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.070230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.069967    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.070702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.070678    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.072543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.072516    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.088583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.088548    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lslnf\" (UniqueName: \"kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf\") pod \"maas-api-revoke-sub-3141e1f6-jhtdd\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.152451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.152413    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:21.158809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.158771    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:21.362462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.362409    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:21.445360 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:21.445322    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d5a96d_dd44_4c20_a261_d338349c6c83.slice/crio-9a7c397d22efb767048dd54b57690d8bce244959f29337942e6bc5ee039776c5 WatchSource:0}: Error finding container 9a7c397d22efb767048dd54b57690d8bce244959f29337942e6bc5ee039776c5: Status 404 returned error can't find the container with id 9a7c397d22efb767048dd54b57690d8bce244959f29337942e6bc5ee039776c5
Nov 04 01:18:21.446973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.446944    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"]
Nov 04 01:18:21.713144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.713097    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd" event={"ID":"29d5a96d-dd44-4c20-a261-d338349c6c83","Type":"ContainerStarted","Data":"9a7c397d22efb767048dd54b57690d8bce244959f29337942e6bc5ee039776c5"}
Nov 04 01:18:21.720013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.715496    5797 generic.go:356] "Generic (PLEG): container finished" podID="85280e39-4b0c-4515-8889-e81e6e77ee3b" containerID="6287560de98dc57e73ae890b17ab42319fa25ffb0920ab338742aa3075943983" exitCode=0
Nov 04 01:18:21.726019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:21.721161    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf" event={"ID":"85280e39-4b0c-4515-8889-e81e6e77ee3b","Type":"ContainerDied","Data":"6287560de98dc57e73ae890b17ab42319fa25ffb0920ab338742aa3075943983"}
Nov 04 01:18:22.199114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.199089    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:22.340067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.333571    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:22.409015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.400575    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca\") pod \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") "
Nov 04 01:18:22.409015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.400701    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca\") pod \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") "
Nov 04 01:18:22.409015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.400901    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn\" (UniqueName: \"kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn\") pod \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\" (UID: \"9521b723-3f93-4e36-8c9c-c415cf6f82e1\") "
Nov 04 01:18:22.451089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.446565    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca" pod "9521b723-3f93-4e36-8c9c-c415cf6f82e1" (UID: "9521b723-3f93-4e36-8c9c-c415cf6f82e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:22.516927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.515139    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:22.517447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.517396    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca" pod "9521b723-3f93-4e36-8c9c-c415cf6f82e1" (UID: "9521b723-3f93-4e36-8c9c-c415cf6f82e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:22.560491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.560443    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn" pod "9521b723-3f93-4e36-8c9c-c415cf6f82e1" (UID: "9521b723-3f93-4e36-8c9c-c415cf6f82e1"). InnerVolumeSpecName "kube-api-access-6whhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:22.618762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.618729    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9521b723-3f93-4e36-8c9c-c415cf6f82e1-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:22.618762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.618759    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whhn\" (UniqueName: \"kubernetes.io/projected/9521b723-3f93-4e36-8c9c-c415cf6f82e1-kube-api-access-6whhn\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:22.725872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.725842    5797 generic.go:356] "Generic (PLEG): container finished" podID="29d5a96d-dd44-4c20-a261-d338349c6c83" containerID="28f680056b5c929ba62aecb3dcfa4f471c8718406e0568742e4580467fc82b27" exitCode=0
Nov 04 01:18:22.726033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.725928    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd" event={"ID":"29d5a96d-dd44-4c20-a261-d338349c6c83","Type":"ContainerDied","Data":"28f680056b5c929ba62aecb3dcfa4f471c8718406e0568742e4580467fc82b27"}
Nov 04 01:18:22.727461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.727439    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"
Nov 04 01:18:22.727564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.727499    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598" event={"ID":"9521b723-3f93-4e36-8c9c-c415cf6f82e1","Type":"ContainerDied","Data":"54fa76960f2a3254701f210c18c1cde7b7b9cf992666064f9e55d219af5a0ac8"}
Nov 04 01:18:22.727564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.727542    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54fa76960f2a3254701f210c18c1cde7b7b9cf992666064f9e55d219af5a0ac8"
Nov 04 01:18:22.892180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:22.892151    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:23.023859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.023774    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca\") pod \"85280e39-4b0c-4515-8889-e81e6e77ee3b\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") "
Nov 04 01:18:23.024196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.024168    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca\") pod \"85280e39-4b0c-4515-8889-e81e6e77ee3b\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") "
Nov 04 01:18:23.024353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.024233    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2\" (UniqueName: \"kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2\") pod \"85280e39-4b0c-4515-8889-e81e6e77ee3b\" (UID: \"85280e39-4b0c-4515-8889-e81e6e77ee3b\") "
Nov 04 01:18:23.024618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.024590    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca" pod "85280e39-4b0c-4515-8889-e81e6e77ee3b" (UID: "85280e39-4b0c-4515-8889-e81e6e77ee3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:23.024948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.024924    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:23.026071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.026045    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca" pod "85280e39-4b0c-4515-8889-e81e6e77ee3b" (UID: "85280e39-4b0c-4515-8889-e81e6e77ee3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:23.026310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.026288    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2" pod "85280e39-4b0c-4515-8889-e81e6e77ee3b" (UID: "85280e39-4b0c-4515-8889-e81e6e77ee3b"). InnerVolumeSpecName "kube-api-access-k26z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:23.126433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.126394    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85280e39-4b0c-4515-8889-e81e6e77ee3b-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:23.126433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.126422    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26z2\" (UniqueName: \"kubernetes.io/projected/85280e39-4b0c-4515-8889-e81e6e77ee3b-kube-api-access-k26z2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:23.211945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.211852    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:23.738090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.738059    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"
Nov 04 01:18:23.738090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.738079    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf" event={"ID":"85280e39-4b0c-4515-8889-e81e6e77ee3b","Type":"ContainerDied","Data":"eaf8c10a75fb4663e2164aef40ad32ea3693a41f446666b8a4cf1ae67d41cf88"}
Nov 04 01:18:23.738320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.738129    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaf8c10a75fb4663e2164aef40ad32ea3693a41f446666b8a4cf1ae67d41cf88"
Nov 04 01:18:23.920894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:23.920869    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:24.010170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.006946    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:24.075297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.075267    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca\") pod \"29d5a96d-dd44-4c20-a261-d338349c6c83\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") "
Nov 04 01:18:24.075463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.075439    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca\") pod \"29d5a96d-dd44-4c20-a261-d338349c6c83\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") "
Nov 04 01:18:24.075583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.075567    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf\" (UniqueName: \"kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf\") pod \"29d5a96d-dd44-4c20-a261-d338349c6c83\" (UID: \"29d5a96d-dd44-4c20-a261-d338349c6c83\") "
Nov 04 01:18:24.075665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.075639    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca" pod "29d5a96d-dd44-4c20-a261-d338349c6c83" (UID: "29d5a96d-dd44-4c20-a261-d338349c6c83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:24.076280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.076249    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:24.077505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.077478    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca" pod "29d5a96d-dd44-4c20-a261-d338349c6c83" (UID: "29d5a96d-dd44-4c20-a261-d338349c6c83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:24.077848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.077825    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf" pod "29d5a96d-dd44-4c20-a261-d338349c6c83" (UID: "29d5a96d-dd44-4c20-a261-d338349c6c83"). InnerVolumeSpecName "kube-api-access-lslnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:24.178071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.178035    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d5a96d-dd44-4c20-a261-d338349c6c83-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:24.178071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.178067    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lslnf\" (UniqueName: \"kubernetes.io/projected/29d5a96d-dd44-4c20-a261-d338349c6c83-kube-api-access-lslnf\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:24.808897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.808758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:24.820027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.819975    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd" event={"ID":"29d5a96d-dd44-4c20-a261-d338349c6c83","Type":"ContainerDied","Data":"9a7c397d22efb767048dd54b57690d8bce244959f29337942e6bc5ee039776c5"}
Nov 04 01:18:24.820184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.820057    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a7c397d22efb767048dd54b57690d8bce244959f29337942e6bc5ee039776c5"
Nov 04 01:18:24.820413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:24.820135    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"
Nov 04 01:18:25.485075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:25.479671    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:25.768838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:25.768744    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:26.183172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.183142    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:26.251571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.251534    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"]
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252559    5797 cpu_manager.go:414] "containerMap: removing container" podUID="29d5a96d-dd44-4c20-a261-d338349c6c83" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252584    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d5a96d-dd44-4c20-a261-d338349c6c83" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252610    5797 cpu_manager.go:414] "containerMap: removing container" podUID="9521b723-3f93-4e36-8c9c-c415cf6f82e1" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252619    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9521b723-3f93-4e36-8c9c-c415cf6f82e1" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252694    5797 cpu_manager.go:414] "containerMap: removing container" podUID="85280e39-4b0c-4515-8889-e81e6e77ee3b" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252703    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85280e39-4b0c-4515-8889-e81e6e77ee3b" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252869    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="85280e39-4b0c-4515-8889-e81e6e77ee3b" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252888    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d5a96d-dd44-4c20-a261-d338349c6c83" containerName="revoke-keys"
Nov 04 01:18:26.255807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.252902    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="9521b723-3f93-4e36-8c9c-c415cf6f82e1" containerName="revoke-keys"
Nov 04 01:18:26.284839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.284301    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.286050 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:26.285968    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:18:26.293896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.293873    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"]
Nov 04 01:18:26.316529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.316498    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.316674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.316546    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kb9n\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.316724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.316693    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.316804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.316745    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.316919 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.316899    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.317018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.316973    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.317083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.317044    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.418983    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.419083    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.419137    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.419344    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.419400    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kb9n\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.419455    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.419524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.419494    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.420292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.420260    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.421752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.421718    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.421752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.421749    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.421923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.421793    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.421923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.421854    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.431582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.431559    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.431844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.431825    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kb9n\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjqs2\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:26.599806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:26.599720    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:27.007459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.002983    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"]
Nov 04 01:18:27.031223 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:27.026343    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f7bd07_f7a6_4817_9644_63ceeff7a207.slice/crio-ff661e098e0124788d4aa56d301fb628f0469ba1a4dae21129c3d167bbefe47d WatchSource:0}: Error finding container ff661e098e0124788d4aa56d301fb628f0469ba1a4dae21129c3d167bbefe47d: Status 404 returned error can't find the container with id ff661e098e0124788d4aa56d301fb628f0469ba1a4dae21129c3d167bbefe47d
Nov 04 01:18:27.038010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.032368    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:27.038010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.032462    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:18:27.284906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.283030    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:27.948826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.948789    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" event={"ID":"f6f7bd07-f7a6-4817-9644-63ceeff7a207","Type":"ContainerStarted","Data":"5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493"}
Nov 04 01:18:27.949049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.948845    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" event={"ID":"f6f7bd07-f7a6-4817-9644-63ceeff7a207","Type":"ContainerStarted","Data":"ff661e098e0124788d4aa56d301fb628f0469ba1a4dae21129c3d167bbefe47d"}
Nov 04 01:18:27.949049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.948868    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:27.988668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:27.985406    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podStartSLOduration=1.985384525 podStartE2EDuration="1.985384525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:27.977786474 +0000 UTC m=+895.619692746" watchObservedRunningTime="2026-11-04 01:18:27.985384525 +0000 UTC m=+895.627290777"
Nov 04 01:18:28.440900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.439283    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:28.638581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.638542    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:28.863084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.863060    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"]
Nov 04 01:18:28.888233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.888202    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:28.892153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.892122    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"]
Nov 04 01:18:28.893245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.893219    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:18:28.893354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.893321    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:18:28.964317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:28.964283    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:18:29.010730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.010690    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.010730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.010735    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk47v\" (UniqueName: \"kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.010952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.010880    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.116091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.116002    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.116091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.116057    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk47v\" (UniqueName: \"kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.116793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.116762    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.116902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.116769    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.124045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.123496    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.130493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.130371    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk47v\" (UniqueName: \"kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v\") pod \"maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.208211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.208176    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:29.308570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.308536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:29.566007 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:29.565963    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60265dc2_8348_475e_83e6_7e2e9fea0d22.slice/crio-8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9 WatchSource:0}: Error finding container 8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9: Status 404 returned error can't find the container with id 8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9
Nov 04 01:18:29.567958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.567930    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"]
Nov 04 01:18:29.707458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.707428    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"]
Nov 04 01:18:29.714172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.714143    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.726382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.723337    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"]
Nov 04 01:18:29.830566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.830441    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.830566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.830533    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.830792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.830771    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggn9r\" (UniqueName: \"kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.932609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.932575    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggn9r\" (UniqueName: \"kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.932814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.932771    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.932814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.932792    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.933570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.933541    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.935348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.935323    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.944485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.944462    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggn9r\" (UniqueName: \"kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r\") pod \"maas-api-revoke-sub-5ba0ba93-cmq5n\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:29.976267 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.976227    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2" event={"ID":"60265dc2-8348-475e-83e6-7e2e9fea0d22","Type":"ContainerStarted","Data":"65c420302873ad657d4dd6ed001726793ac7d15a344f7c8242de7c9bce710fd8"}
Nov 04 01:18:29.976418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:29.976292    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2" event={"ID":"60265dc2-8348-475e-83e6-7e2e9fea0d22","Type":"ContainerStarted","Data":"8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9"}
Nov 04 01:18:30.072975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:30.058159    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:30.087613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:30.076085    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2" podStartSLOduration=2.076064701 podStartE2EDuration="2.076064701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:30.012416982 +0000 UTC m=+897.654323233" watchObservedRunningTime="2026-11-04 01:18:30.076064701 +0000 UTC m=+897.717970987"
Nov 04 01:18:30.113015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:30.108983    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:30.355920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:30.354977    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:30.567246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:30.567209    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:30.599291 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:30.599258    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a88e516_f86a_451f_9645_619b26d1eb86.slice/crio-5225b491b6b9921f4e21a72d76c07c909db423e07db7d9b80f5a0d97384f6125 WatchSource:0}: Error finding container 5225b491b6b9921f4e21a72d76c07c909db423e07db7d9b80f5a0d97384f6125: Status 404 returned error can't find the container with id 5225b491b6b9921f4e21a72d76c07c909db423e07db7d9b80f5a0d97384f6125
Nov 04 01:18:30.602078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:30.601981    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"]
Nov 04 01:18:31.023127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:31.021857    5797 generic.go:356] "Generic (PLEG): container finished" podID="60265dc2-8348-475e-83e6-7e2e9fea0d22" containerID="65c420302873ad657d4dd6ed001726793ac7d15a344f7c8242de7c9bce710fd8" exitCode=0
Nov 04 01:18:31.023127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:31.022035    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2" event={"ID":"60265dc2-8348-475e-83e6-7e2e9fea0d22","Type":"ContainerDied","Data":"65c420302873ad657d4dd6ed001726793ac7d15a344f7c8242de7c9bce710fd8"}
Nov 04 01:18:31.098566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:31.091439    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n" event={"ID":"6a88e516-f86a-451f-9645-619b26d1eb86","Type":"ContainerStarted","Data":"2a6a4fe5e10cc7423a5f5525a1dc4f282049071026cb5b1939ed86d62573420c"}
Nov 04 01:18:31.098566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:31.091509    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n" event={"ID":"6a88e516-f86a-451f-9645-619b26d1eb86","Type":"ContainerStarted","Data":"5225b491b6b9921f4e21a72d76c07c909db423e07db7d9b80f5a0d97384f6125"}
Nov 04 01:18:31.248358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:31.242450    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n" podStartSLOduration=2.2424274 podStartE2EDuration="2.2424274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:31.149813099 +0000 UTC m=+898.791719349" watchObservedRunningTime="2026-11-04 01:18:31.2424274 +0000 UTC m=+898.884333650"
Nov 04 01:18:31.593111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:31.593023    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:32.182015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.181234    5797 generic.go:356] "Generic (PLEG): container finished" podID="6a88e516-f86a-451f-9645-619b26d1eb86" containerID="2a6a4fe5e10cc7423a5f5525a1dc4f282049071026cb5b1939ed86d62573420c" exitCode=0
Nov 04 01:18:32.182015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.181688    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n" event={"ID":"6a88e516-f86a-451f-9645-619b26d1eb86","Type":"ContainerDied","Data":"2a6a4fe5e10cc7423a5f5525a1dc4f282049071026cb5b1939ed86d62573420c"}
Nov 04 01:18:32.265021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.260666    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:32.352013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.349907    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:32.362078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.361463    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:32.782015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.778341    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:32.799339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.783270    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:32.847018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:32.806281    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:33.074071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:33.056080    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:33.088016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:33.079373    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:33.088016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:33.079698    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:33.331369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:33.331272    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:34.077696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:33.379343    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:34.147036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.128377    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:34.147036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.130881    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:18:34.193201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.193143    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:34.240864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.240818    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:34.305011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.267538    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:34.611196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.611154    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:34.611720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.611695    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:18:34.735936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.658064    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:34.833256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.833195    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:34.881375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:34.131495    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.152131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.147837    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:35.193277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.193234    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.274809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.196077    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:35.303198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.213606    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.316024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.242083    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:35.316242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.303527    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.324043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.304076    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.533833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.533227    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.765801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.765756    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:35.823118    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.109s"
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.823236    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.823283    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.823317    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.823354    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.823390    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:18:35.825080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.823426    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:18:35.895030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.892736    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.931425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.931369    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:35.932808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:35.932783    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:18:36.012142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.012092    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:36.101197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.093734    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:36.122014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.109242    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:36.223016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.222588    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca\") pod \"60265dc2-8348-475e-83e6-7e2e9fea0d22\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") "
Nov 04 01:18:36.223016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.222722    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca\") pod \"60265dc2-8348-475e-83e6-7e2e9fea0d22\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") "
Nov 04 01:18:36.223016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.222982    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v\" (UniqueName: \"kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v\") pod \"60265dc2-8348-475e-83e6-7e2e9fea0d22\" (UID: \"60265dc2-8348-475e-83e6-7e2e9fea0d22\") "
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.245134    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.245188    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.251350    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca" pod "60265dc2-8348-475e-83e6-7e2e9fea0d22" (UID: "60265dc2-8348-475e-83e6-7e2e9fea0d22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.269910    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.269977    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.318828    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.319247    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.319548    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.319807    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:36.337026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.326605    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:36.569208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.569147    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:36.905785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.895184    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v" pod "60265dc2-8348-475e-83e6-7e2e9fea0d22" (UID: "60265dc2-8348-475e-83e6-7e2e9fea0d22"). InnerVolumeSpecName "kube-api-access-sk47v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:36.905785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.903369    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:36.905785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.905732    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk47v\" (UniqueName: \"kubernetes.io/projected/60265dc2-8348-475e-83e6-7e2e9fea0d22-kube-api-access-sk47v\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:36.925656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.925603    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:36.925969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:36.925925    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:37.108033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.103081    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:37.144039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.137040    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:37.208072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.144858    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca" pod "60265dc2-8348-475e-83e6-7e2e9fea0d22" (UID: "60265dc2-8348-475e-83e6-7e2e9fea0d22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:37.259021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.248039    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60265dc2-8348-475e-83e6-7e2e9fea0d22-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:37.283337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.283291    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:37.433024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.362595    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:37.452260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.394157    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:37.461969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.394218    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:37.500146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.394241    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:37.631538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.631509    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"
Nov 04 01:18:37.663144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.663088    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:37.747089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.735772    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2" event={"ID":"60265dc2-8348-475e-83e6-7e2e9fea0d22","Type":"ContainerDied","Data":"8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9"}
Nov 04 01:18:37.747089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.735852    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9"
Nov 04 01:18:37.880691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.880639    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 04 01:18:37.920331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:37.920286    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:38.247288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.247216    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:38.247584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.247538    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:38.298410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.298353    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:18:38.472442 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:38.458637    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60265dc2_8348_475e_83e6_7e2e9fea0d22.slice/crio-8d183b0de17b4fc9b0b7088a30f115c367b3738ee3e4ef02611b800a4fbe98e9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60265dc2_8348_475e_83e6_7e2e9fea0d22.slice\": RecentStats: unable to find data in memory cache]"
Nov 04 01:18:38.650379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.650315    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:38.675651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.674143    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:38.675651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.674723    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:38.675651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.674895    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:38.675651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.675208    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:38.675651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.675317    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:18:38.973915 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:38.973800    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:39.310357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.310275    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:18:39.773016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.772305    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n" event={"ID":"6a88e516-f86a-451f-9645-619b26d1eb86","Type":"ContainerDied","Data":"5225b491b6b9921f4e21a72d76c07c909db423e07db7d9b80f5a0d97384f6125"}
Nov 04 01:18:39.773016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.772372    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5225b491b6b9921f4e21a72d76c07c909db423e07db7d9b80f5a0d97384f6125"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.633561    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.382351    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.676222    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.664403    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.676241    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.676279    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:39.783036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.676165    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:39.804893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.804864    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"]
Nov 04 01:18:39.806415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.806388    5797 cpu_manager.go:414] "containerMap: removing container" podUID="6a88e516-f86a-451f-9645-619b26d1eb86" containerName="revoke-keys"
Nov 04 01:18:39.806614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.806598    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a88e516-f86a-451f-9645-619b26d1eb86" containerName="revoke-keys"
Nov 04 01:18:39.806823 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.806808    5797 cpu_manager.go:414] "containerMap: removing container" podUID="60265dc2-8348-475e-83e6-7e2e9fea0d22" containerName="revoke-keys"
Nov 04 01:18:39.806945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.806931    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60265dc2-8348-475e-83e6-7e2e9fea0d22" containerName="revoke-keys"
Nov 04 01:18:39.821393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.821360    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="60265dc2-8348-475e-83e6-7e2e9fea0d22" containerName="revoke-keys"
Nov 04 01:18:39.821671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.821652    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a88e516-f86a-451f-9645-619b26d1eb86" containerName="revoke-keys"
Nov 04 01:18:39.868353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.868319    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"]
Nov 04 01:18:39.868765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.868744    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:39.884082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.884046    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca\") pod \"6a88e516-f86a-451f-9645-619b26d1eb86\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") "
Nov 04 01:18:39.884477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.884454    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca\") pod \"6a88e516-f86a-451f-9645-619b26d1eb86\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") "
Nov 04 01:18:39.884720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.884703    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r\" (UniqueName: \"kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r\") pod \"6a88e516-f86a-451f-9645-619b26d1eb86\" (UID: \"6a88e516-f86a-451f-9645-619b26d1eb86\") "
Nov 04 01:18:39.962016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.952229    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca" pod "6a88e516-f86a-451f-9645-619b26d1eb86" (UID: "6a88e516-f86a-451f-9645-619b26d1eb86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:39.985410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.983524    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca" pod "6a88e516-f86a-451f-9645-619b26d1eb86" (UID: "6a88e516-f86a-451f-9645-619b26d1eb86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:39.985632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.977835    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:40.091414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:39.986979    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77m86\" (UniqueName: \"kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.111906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.111867    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.112215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.112188    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.112623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.112602    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:40.112751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.112736    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a88e516-f86a-451f-9645-619b26d1eb86-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:40.112892 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.016601    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r" pod "6a88e516-f86a-451f-9645-619b26d1eb86" (UID: "6a88e516-f86a-451f-9645-619b26d1eb86"). InnerVolumeSpecName "kube-api-access-ggn9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:40.343292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.229984    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77m86\" (UniqueName: \"kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.372946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.372902    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.373218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.373194    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.373580 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.373559    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggn9r\" (UniqueName: \"kubernetes.io/projected/6a88e516-f86a-451f-9645-619b26d1eb86-kube-api-access-ggn9r\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:40.374507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.374481    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.374639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.280803    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.522677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.496527    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:40.523075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.522954    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.523505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.510252    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.673508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.660777    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77m86\" (UniqueName: \"kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.777016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.776607    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:40.808971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.808931    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-383a864b-kwphq\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.846024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.839185    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:18:40.846805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.846764    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.848297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.848269    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:40.849047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.848792    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:40.861021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.849423    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:40.879023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.849959    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:40.883014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.879280    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.888091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.876872    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"
Nov 04 01:18:40.938546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.938363    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:40.938546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.938423    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.942013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.940958    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:40.994371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.989272    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:40.994371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.989426    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:40.994371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:40.989627    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:41.408020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.396973    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.408020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.397503    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:18:41.779189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.779141    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:41.819116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.813762    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:41.819116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.815281    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.853579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.839596    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.857943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.854438    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.857943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.855727    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:18:41.857943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.856619    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.857943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.857559    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.869897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.868447    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.903853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.903810    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:41.904132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.904101    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:41.995016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:41.990763    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:18:42.039083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:42.034828    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:42.280553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:42.280513    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:42.280803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:42.280774    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:42.361790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:42.357685    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:42.455705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:42.399960    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:42.880172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:42.880111    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:43.143089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.021412    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:18:43.161039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.045383    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:43.183891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.183576    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:43.255038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.252607    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:18:43.256912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.256855    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:43.376381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.338534    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:43.540031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.529382    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:43.547039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.543175    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:43.754073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:43.729019    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:44.195288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.192865    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:44.259114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.248540    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:18:44.259114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.248962    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.347019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.343324    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.376644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.375188    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.435446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.434022    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.471063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.468662    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:44.471063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.468725    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.493273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.491507    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:44.544162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.544108    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.617138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.617095    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.633396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.630143    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.839076 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:44.838310    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.322s"
Nov 04 01:18:44.839076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.838418    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:18:44.839076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.838659    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:18:44.877083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.841189    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:44.996121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.992722    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq" event={"ID":"be0cad38-395d-458e-a6f2-b5597a48e79b","Type":"ContainerStarted","Data":"72f19054e4d7710729cbea47e6c7d8359874b702ae45f04ed82bdef8c95186d9"}
Nov 04 01:18:44.996121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:44.992797    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"]
Nov 04 01:18:45.172018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.165255    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:18:45.219732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.219706    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:18:45.233322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.233285    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:18:45.247125 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.247085    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:18:45.251952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.251903    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:45.265382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.265336    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:45.395413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.395378    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:18:45.397331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.397308    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:18:45.429064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.422520    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:18:45.846051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.844328    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:45.996056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:45.995967    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq" event={"ID":"be0cad38-395d-458e-a6f2-b5597a48e79b","Type":"ContainerStarted","Data":"829a8f305ab40043cc9c84824cbed955584a47047991c81badb01ac30ecfcd3f"}
Nov 04 01:18:46.021537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:46.021487    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq" podStartSLOduration=8.021469269 podStartE2EDuration="8.021469269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:46.016201657 +0000 UTC m=+913.658107932" watchObservedRunningTime="2026-11-04 01:18:46.021469269 +0000 UTC m=+913.663375518"
Nov 04 01:18:46.820069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:46.819613    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:47.007462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.007428    5797 generic.go:356] "Generic (PLEG): container finished" podID="be0cad38-395d-458e-a6f2-b5597a48e79b" containerID="829a8f305ab40043cc9c84824cbed955584a47047991c81badb01ac30ecfcd3f" exitCode=0
Nov 04 01:18:47.007875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.007513    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq" event={"ID":"be0cad38-395d-458e-a6f2-b5597a48e79b","Type":"ContainerDied","Data":"829a8f305ab40043cc9c84824cbed955584a47047991c81badb01ac30ecfcd3f"}
Nov 04 01:18:47.709017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.698843    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:47.807211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.807155    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"]
Nov 04 01:18:47.826499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.826465    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"]
Nov 04 01:18:47.826717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.826598    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:47.926067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.926038    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:47.926285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.926270    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4x6\" (UniqueName: \"kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:47.926426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:47.926399    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.027970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.027933    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.028404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.027979    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4x6\" (UniqueName: \"kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.028404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.028049    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.029589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.029555    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.033048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.031405    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.059476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.047558    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4x6\" (UniqueName: \"kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6\") pod \"maas-api-revoke-keys-e2e-ait-8f95141e-d998t\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.147046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.144417    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:48.322505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.322133    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:48.434859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.434826    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca\") pod \"be0cad38-395d-458e-a6f2-b5597a48e79b\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") "
Nov 04 01:18:48.435081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.434913    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86\" (UniqueName: \"kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86\") pod \"be0cad38-395d-458e-a6f2-b5597a48e79b\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") "
Nov 04 01:18:48.435185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.435168    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca\") pod \"be0cad38-395d-458e-a6f2-b5597a48e79b\" (UID: \"be0cad38-395d-458e-a6f2-b5597a48e79b\") "
Nov 04 01:18:48.436876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.436846    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca" pod "be0cad38-395d-458e-a6f2-b5597a48e79b" (UID: "be0cad38-395d-458e-a6f2-b5597a48e79b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:48.439839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.439813    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86" pod "be0cad38-395d-458e-a6f2-b5597a48e79b" (UID: "be0cad38-395d-458e-a6f2-b5597a48e79b"). InnerVolumeSpecName "kube-api-access-77m86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:48.441798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.441774    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca" pod "be0cad38-395d-458e-a6f2-b5597a48e79b" (UID: "be0cad38-395d-458e-a6f2-b5597a48e79b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:48.537175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.536659    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:48.537175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.536684    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77m86\" (UniqueName: \"kubernetes.io/projected/be0cad38-395d-458e-a6f2-b5597a48e79b-kube-api-access-77m86\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:48.537175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.536694    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be0cad38-395d-458e-a6f2-b5597a48e79b-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:48.540310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.540283    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"]
Nov 04 01:18:48.540628 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:18:48.540602    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa79e873_80f1_4cd7_b833_2b1697c50c71.slice/crio-eca16b5fd1587c2c2eafffdf5d3c8ad59d8d3dcbf20b17c8de4d788db56045e2 WatchSource:0}: Error finding container eca16b5fd1587c2c2eafffdf5d3c8ad59d8d3dcbf20b17c8de4d788db56045e2: Status 404 returned error can't find the container with id eca16b5fd1587c2c2eafffdf5d3c8ad59d8d3dcbf20b17c8de4d788db56045e2
Nov 04 01:18:48.945420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:48.936533    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:49.034683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:49.034652    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"
Nov 04 01:18:49.035195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:49.034654    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq" event={"ID":"be0cad38-395d-458e-a6f2-b5597a48e79b","Type":"ContainerDied","Data":"72f19054e4d7710729cbea47e6c7d8359874b702ae45f04ed82bdef8c95186d9"}
Nov 04 01:18:49.035195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:49.035172    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f19054e4d7710729cbea47e6c7d8359874b702ae45f04ed82bdef8c95186d9"
Nov 04 01:18:49.037222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:49.036849    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t" event={"ID":"fa79e873-80f1-4cd7-b833-2b1697c50c71","Type":"ContainerStarted","Data":"eca16b5fd1587c2c2eafffdf5d3c8ad59d8d3dcbf20b17c8de4d788db56045e2"}
Nov 04 01:18:49.887050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:49.885788    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:50.097012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.092411    5797 generic.go:356] "Generic (PLEG): container finished" podID="fa79e873-80f1-4cd7-b833-2b1697c50c71" containerID="d329b2d9b88e01f9b0ea176d5051ac88adb8442e8679192b24c65c98e86292e7" exitCode=0
Nov 04 01:18:50.097012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.092503    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t" event={"ID":"fa79e873-80f1-4cd7-b833-2b1697c50c71","Type":"ContainerDied","Data":"d329b2d9b88e01f9b0ea176d5051ac88adb8442e8679192b24c65c98e86292e7"}
Nov 04 01:18:50.097012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.093644    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:50.316530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.316440    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:50.325138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.325072    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"]
Nov 04 01:18:50.335856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.335720    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cc22d46-kq8g2"]
Nov 04 01:18:50.477017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.472857    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"]
Nov 04 01:18:50.477017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.473122    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" podUID="0039b909-973d-4845-bb8d-db4dd4c57445" containerName="istio-proxy" containerID="cri-o://406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3" gracePeriod=30
Nov 04 01:18:50.523012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.521242    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60265dc2-8348-475e-83e6-7e2e9fea0d22" path="/var/lib/kubelet/pods/60265dc2-8348-475e-83e6-7e2e9fea0d22/volumes"
Nov 04 01:18:50.639010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.636470    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:50.865560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:50.847257    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:51.409023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.406467    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:51.625709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.625678    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:51.704494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.704413    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:51.705563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.705539    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6\" (UniqueName: \"kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6\") pod \"fa79e873-80f1-4cd7-b833-2b1697c50c71\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") "
Nov 04 01:18:51.705747 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.705606    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca\") pod \"fa79e873-80f1-4cd7-b833-2b1697c50c71\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") "
Nov 04 01:18:51.705956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.705935    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca\") pod \"fa79e873-80f1-4cd7-b833-2b1697c50c71\" (UID: \"fa79e873-80f1-4cd7-b833-2b1697c50c71\") "
Nov 04 01:18:51.706536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.706484    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca" pod "fa79e873-80f1-4cd7-b833-2b1697c50c71" (UID: "fa79e873-80f1-4cd7-b833-2b1697c50c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:51.707584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.707525    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:51.708940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.708913    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6" pod "fa79e873-80f1-4cd7-b833-2b1697c50c71" (UID: "fa79e873-80f1-4cd7-b833-2b1697c50c71"). InnerVolumeSpecName "kube-api-access-ks4x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:51.709427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.709403    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca" pod "fa79e873-80f1-4cd7-b833-2b1697c50c71" (UID: "fa79e873-80f1-4cd7-b833-2b1697c50c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:18:51.809680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.809644    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4x6\" (UniqueName: \"kubernetes.io/projected/fa79e873-80f1-4cd7-b833-2b1697c50c71-kube-api-access-ks4x6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:51.809680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:51.809683    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa79e873-80f1-4cd7-b833-2b1697c50c71-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:52.153568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:52.153522    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t" event={"ID":"fa79e873-80f1-4cd7-b833-2b1697c50c71","Type":"ContainerDied","Data":"eca16b5fd1587c2c2eafffdf5d3c8ad59d8d3dcbf20b17c8de4d788db56045e2"}
Nov 04 01:18:52.159300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:52.159268    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eca16b5fd1587c2c2eafffdf5d3c8ad59d8d3dcbf20b17c8de4d788db56045e2"
Nov 04 01:18:52.159619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:52.159600    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"
Nov 04 01:18:52.361024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:52.359536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:52.839179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:52.839134    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:53.155503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:53.153478    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:53.304185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:53.304142    5797 patch_prober.go:29] interesting pod/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:18:53.304346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:53.304230    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" podUID="0039b909-973d-4845-bb8d-db4dd4c57445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:18:53.383730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:53.383698    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:53.596470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:53.596374    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:53.827071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:53.827031    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:54.557218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:54.557177    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:55.046776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:55.046732    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:55.272101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:55.272062    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:55.475320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:55.475244    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:56.008467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:56.008426    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:56.253967 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:56.253928    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:18:56.780594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:56.780562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:57.098629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.098602    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:57.205575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.205541    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.205773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.205659    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.205773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.205704    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.205773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.205767    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.205948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.205866    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.206017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.205968    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:18:57.206017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206008    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.206127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206048    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:18:57.206127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206094    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.206255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206170    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.206255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206207    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data\") pod \"0039b909-973d-4845-bb8d-db4dd4c57445\" (UID: \"0039b909-973d-4845-bb8d-db4dd4c57445\") "
Nov 04 01:18:57.206460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206419    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206551    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206578    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.206969    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.207024    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.207035    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.207045    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039b909-973d-4845-bb8d-db4dd4c57445-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.207090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.207055    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.208578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208550    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:18:57.208734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208600    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:57.208734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208689    5797 generic.go:356] "Generic (PLEG): container finished" podID="0039b909-973d-4845-bb8d-db4dd4c57445" containerID="406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3" exitCode=0
Nov 04 01:18:57.208734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208710    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:18:57.208872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208755    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"
Nov 04 01:18:57.208872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208759    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" event={"ID":"0039b909-973d-4845-bb8d-db4dd4c57445","Type":"ContainerDied","Data":"406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3"}
Nov 04 01:18:57.208872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208814    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs" event={"ID":"0039b909-973d-4845-bb8d-db4dd4c57445","Type":"ContainerDied","Data":"769dabfd31dc0074fbebdfe455edf06c2ad658ff8b98143b95ce503bb81bcd2e"}
Nov 04 01:18:57.208872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.208841    5797 scope.go:122] "RemoveContainer" containerID="406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3"
Nov 04 01:18:57.209171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.209144    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql" pod "0039b909-973d-4845-bb8d-db4dd4c57445" (UID: "0039b909-973d-4845-bb8d-db4dd4c57445"). InnerVolumeSpecName "kube-api-access-zwcql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:18:57.249130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.245863    5797 scope.go:122] "RemoveContainer" containerID="406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3"
Nov 04 01:18:57.249130 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:18:57.246315    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3\": container with ID starting with 406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3 not found: ID does not exist" containerID="406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3"
Nov 04 01:18:57.249130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.246353    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3"} err="failed to get container status \"406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3\": rpc error: code = NotFound desc = could not find container \"406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3\": container with ID starting with 406dbf39d946610929633f103cbccda78646719235166d4f0157f6b0b9aabde3 not found: ID does not exist"
Nov 04 01:18:57.308541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.308513    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.308541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.308540    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039b909-973d-4845-bb8d-db4dd4c57445-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.308731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.308551    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039b909-973d-4845-bb8d-db4dd4c57445-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.308731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.308563    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcql\" (UniqueName: \"kubernetes.io/projected/0039b909-973d-4845-bb8d-db4dd4c57445-kube-api-access-zwcql\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:18:57.580880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.580842    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:57.609185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.609061    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"]
Nov 04 01:18:57.619166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:57.619137    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9cc22d46-openshift-default-5485d8bb95-zm7hs"]
Nov 04 01:18:58.088595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:58.088558    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:58.289822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:58.289774    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:58.490078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:58.490043    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0039b909-973d-4845-bb8d-db4dd4c57445" path="/var/lib/kubelet/pods/0039b909-973d-4845-bb8d-db4dd4c57445/volumes"
Nov 04 01:18:58.990393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:58.990359    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:59.198024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:59.193563    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:18:59.866835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:18:59.866791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:00.585332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:00.585293    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:00.888705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:00.888665    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:01.215567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:01.215464    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:01.486895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:01.486785    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:01.839771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:01.839590    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:02.049231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:02.049164    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:02.545722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:02.545684    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:03.289251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:03.289211    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:03.495393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:03.495360    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:03.694960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:03.694914    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:04.184655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:04.184620    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:05.019748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.019711    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:05.174297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.174252    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"]
Nov 04 01:19:05.179960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.179934    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383a864b-kwphq"]
Nov 04 01:19:05.221706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.221670    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:05.453549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.453510    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:05.677704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.677668    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:05.678383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.678364    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"]
Nov 04 01:19:05.687232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.687199    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f95141e-d998t"]
Nov 04 01:19:05.877212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:05.877175    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:06.184549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.184469    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:06.402616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.402579    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:06.482801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.482718    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0cad38-395d-458e-a6f2-b5597a48e79b" path="/var/lib/kubelet/pods/be0cad38-395d-458e-a6f2-b5597a48e79b/volumes"
Nov 04 01:19:06.483375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.483350    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa79e873-80f1-4cd7-b833-2b1697c50c71" path="/var/lib/kubelet/pods/fa79e873-80f1-4cd7-b833-2b1697c50c71/volumes"
Nov 04 01:19:06.749059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.748955    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:06.939294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.939258    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"]
Nov 04 01:19:06.939555 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.939518    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" podUID="69fb7b17-2b56-486a-8c75-051f82dd7045" containerName="istio-proxy" containerID="cri-o://20f029a846e7c28edd1260f7e501b5487b8dfb4fa05ec73860924c79433c310e" gracePeriod=30
Nov 04 01:19:06.994304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:06.994271    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:07.210065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:07.205055    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:07.407591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:07.407559    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:07.710313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:07.710278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:07.912303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:07.912275    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:08.123883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:08.123843    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:08.336785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:08.336753    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:08.953967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:08.953883    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:09.159780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:09.159751    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:09.963815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:09.963765    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:10.307536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:10.307440    5797 patch_prober.go:29] interesting pod/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:19:10.307701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:10.307526    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" podUID="69fb7b17-2b56-486a-8c75-051f82dd7045" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:19:10.619058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:10.619020    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:10.831593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:10.831557    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:11.047424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:11.047342    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:12.164070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:12.158004    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:12.572597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:12.572553    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:12.967014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:12.960921    5797 generic.go:356] "Generic (PLEG): container finished" podID="69fb7b17-2b56-486a-8c75-051f82dd7045" containerID="20f029a846e7c28edd1260f7e501b5487b8dfb4fa05ec73860924c79433c310e" exitCode=0
Nov 04 01:19:12.967014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:12.961027    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" event={"ID":"69fb7b17-2b56-486a-8c75-051f82dd7045","Type":"ContainerDied","Data":"20f029a846e7c28edd1260f7e501b5487b8dfb4fa05ec73860924c79433c310e"}
Nov 04 01:19:13.002029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.000266    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:13.054982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.054940    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:19:13.184020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.180614    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.212409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.212315    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.212726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.212703    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.213018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.212999    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.213243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.213227    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.213426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.213410    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.213617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.213601    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.213743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.213729    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.230010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.213920    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert\") pod \"69fb7b17-2b56-486a-8c75-051f82dd7045\" (UID: \"69fb7b17-2b56-486a-8c75-051f82dd7045\") "
Nov 04 01:19:13.235495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.234401    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:19:13.235495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.235274    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:19:13.236107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.235790    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:19:13.236803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.236781    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:19:13.262075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.249088    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:19:13.272463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.272403    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:19:13.272827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.272794    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:19:13.273155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.273135    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "kube-api-access-n882d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:19:13.273914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.273879    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token" pod "69fb7b17-2b56-486a-8c75-051f82dd7045" (UID: "69fb7b17-2b56-486a-8c75-051f82dd7045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336342    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336381    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336396    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336413    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n882d\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-kube-api-access-n882d\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336429    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336444    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336461    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb7b17-2b56-486a-8c75-051f82dd7045-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336476    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7b17-2b56-486a-8c75-051f82dd7045-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:13.336516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:13.336490    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb7b17-2b56-486a-8c75-051f82dd7045-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:14.013181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.013052    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5" event={"ID":"69fb7b17-2b56-486a-8c75-051f82dd7045","Type":"ContainerDied","Data":"68337384be52d7134414cad2a706c632d0e282c49b3341b7dab12712e58cecaa"}
Nov 04 01:19:14.020128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.013472    5797 scope.go:122] "RemoveContainer" containerID="20f029a846e7c28edd1260f7e501b5487b8dfb4fa05ec73860924c79433c310e"
Nov 04 01:19:14.021118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.020506    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"
Nov 04 01:19:14.644753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.644702    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:14.726016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.718650    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:14.753209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.751538    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:19:14.754516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.754473    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:19:14.914090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:14.913969    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:19:15.341004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.335350    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:19:15.341004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.338612    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:15.418020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.412664    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:15.436016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.424731    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:15.834223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.821070    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:15.913162 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:19:15.912628    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.389s"
Nov 04 01:19:15.913162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.912793    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:19:15.913162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.912822    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"]
Nov 04 01:19:15.936427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.933029    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:15.944651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.939772    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:19:15.960245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.949303    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:15.960245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.949637    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:19:15.964536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.964497    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:15.969014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:15.964769    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:19:16.217123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.217083    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-383a864b-openshift-default-56dd98bf4c-x6sv5"]
Nov 04 01:19:16.440575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.440529    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:16.466016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.441634    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:19:16.819186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.819138    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:19:16.873011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.854789    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:16.961015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.936495    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:16.979083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.961837    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:19:16.983014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:16.965627    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:19:17.108055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.107633    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fb7b17-2b56-486a-8c75-051f82dd7045" path="/var/lib/kubelet/pods/69fb7b17-2b56-486a-8c75-051f82dd7045/volumes"
Nov 04 01:19:17.322399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.322357    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:19:17.358220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.358105    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:17.451370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.451327    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:17.517375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.510803    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:17.758047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.750571    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:17.870454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.869814    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:19:17.870454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.870332    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:19:17.878334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.877193    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:19:17.994108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.994059    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:19:18.008112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.994441    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:19:18.050314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:17.997657    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:18.336133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.336026    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:19:18.337873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.337849    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:19:18.497254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.497208    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:18.500775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.500742    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:18.517468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.517436    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:19:18.880538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.880481    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:19:18.885456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:18.885040    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:19.104015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:19.006558    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:19.104015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:19.029345    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:19.104015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:19.056101    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:19:19.349181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:19.349071    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:19:19.719012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:19.642166    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:19:19.719388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:19.642216    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:19:20.034668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.034554    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:19:20.252678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.252626    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:19:20.296051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.269328    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:19:20.406678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.406628    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:19:20.459864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.459834    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:19:20.498011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.473678    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:20.515017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.475122    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:19:20.518065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.515353    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:19:20.534028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.533806    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:19:20.708058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.708020    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"]
Nov 04 01:19:20.709104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709077    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0039b909-973d-4845-bb8d-db4dd4c57445" containerName="istio-proxy"
Nov 04 01:19:20.709104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709104    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0039b909-973d-4845-bb8d-db4dd4c57445" containerName="istio-proxy"
Nov 04 01:19:20.709292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709126    5797 cpu_manager.go:414] "containerMap: removing container" podUID="69fb7b17-2b56-486a-8c75-051f82dd7045" containerName="istio-proxy"
Nov 04 01:19:20.709292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709134    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fb7b17-2b56-486a-8c75-051f82dd7045" containerName="istio-proxy"
Nov 04 01:19:20.709292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709175    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fa79e873-80f1-4cd7-b833-2b1697c50c71" containerName="revoke-keys"
Nov 04 01:19:20.709292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709183    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa79e873-80f1-4cd7-b833-2b1697c50c71" containerName="revoke-keys"
Nov 04 01:19:20.709292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709227    5797 cpu_manager.go:414] "containerMap: removing container" podUID="be0cad38-395d-458e-a6f2-b5597a48e79b" containerName="revoke-keys"
Nov 04 01:19:20.709292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709235    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0cad38-395d-458e-a6f2-b5597a48e79b" containerName="revoke-keys"
Nov 04 01:19:20.709613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709404    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa79e873-80f1-4cd7-b833-2b1697c50c71" containerName="revoke-keys"
Nov 04 01:19:20.709613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709424    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0cad38-395d-458e-a6f2-b5597a48e79b" containerName="revoke-keys"
Nov 04 01:19:20.709613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709439    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0039b909-973d-4845-bb8d-db4dd4c57445" containerName="istio-proxy"
Nov 04 01:19:20.709613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.709449    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fb7b17-2b56-486a-8c75-051f82dd7045" containerName="istio-proxy"
Nov 04 01:19:20.716525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.716445    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:19:20.716649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.716536    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:19:20.716649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.716565    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"]
Nov 04 01:19:20.716756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.716663    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:19:20.716756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.716708    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:19:20.717473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.717451    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.725424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.724301    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:19:20.725424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.724454    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:19:20.747145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.747114    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:20.831506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.831403    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.831506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.831452    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.832454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.831564    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krhcm\" (UniqueName: \"kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.932873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.932824    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krhcm\" (UniqueName: \"kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.933266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.933245    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.933367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.933282    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.934151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.934123    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.946331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.946297    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krhcm\" (UniqueName: \"kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:20.948205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:20.948177    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cca527b7-pz947\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:21.053754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:21.053672    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:21.376759 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:19:21.376730    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bbabd0_544f_4124_9b79_3feb723f2952.slice/crio-5f74c83020e68840f2ee73c78443b099dd98a35a79b9f898a35c3730be4e77ad WatchSource:0}: Error finding container 5f74c83020e68840f2ee73c78443b099dd98a35a79b9f898a35c3730be4e77ad: Status 404 returned error can't find the container with id 5f74c83020e68840f2ee73c78443b099dd98a35a79b9f898a35c3730be4e77ad
Nov 04 01:19:21.379364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:21.379311    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"]
Nov 04 01:19:21.511233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:21.511192    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:21.672061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:21.658434    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947" event={"ID":"07bbabd0-544f-4124-9b79-3feb723f2952","Type":"ContainerStarted","Data":"5f74c83020e68840f2ee73c78443b099dd98a35a79b9f898a35c3730be4e77ad"}
Nov 04 01:19:22.466140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:22.454889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:22.668966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:22.668881    5797 generic.go:356] "Generic (PLEG): container finished" podID="07bbabd0-544f-4124-9b79-3feb723f2952" containerID="bba19a1f37a9b93fab16783c72838adee7221aef4dc8d2d8aecfffdd7a14acaa" exitCode=0
Nov 04 01:19:22.669140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:22.668971    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947" event={"ID":"07bbabd0-544f-4124-9b79-3feb723f2952","Type":"ContainerDied","Data":"bba19a1f37a9b93fab16783c72838adee7221aef4dc8d2d8aecfffdd7a14acaa"}
Nov 04 01:19:23.037871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:23.037787    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"]
Nov 04 01:19:23.051852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:23.051716    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"]
Nov 04 01:19:23.061025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:23.060974    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4986295-nhm6f"]
Nov 04 01:19:23.067333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:23.067303    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0643cd6-sdjnf"]
Nov 04 01:19:23.210981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:23.210937    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:23.866596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:23.866572    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:24.002183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.002050    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:24.014217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.013739    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca\") pod \"07bbabd0-544f-4124-9b79-3feb723f2952\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") "
Nov 04 01:19:24.014731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.014710    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm\" (UniqueName: \"kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm\") pod \"07bbabd0-544f-4124-9b79-3feb723f2952\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") "
Nov 04 01:19:24.014856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.014746    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca\") pod \"07bbabd0-544f-4124-9b79-3feb723f2952\" (UID: \"07bbabd0-544f-4124-9b79-3feb723f2952\") "
Nov 04 01:19:24.015359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.015324    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca" pod "07bbabd0-544f-4124-9b79-3feb723f2952" (UID: "07bbabd0-544f-4124-9b79-3feb723f2952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:19:24.015802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.015779    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:24.017203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.017098    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca" pod "07bbabd0-544f-4124-9b79-3feb723f2952" (UID: "07bbabd0-544f-4124-9b79-3feb723f2952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:19:24.020563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.020532    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm" pod "07bbabd0-544f-4124-9b79-3feb723f2952" (UID: "07bbabd0-544f-4124-9b79-3feb723f2952"). InnerVolumeSpecName "kube-api-access-krhcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:19:24.117917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.117881    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bbabd0-544f-4124-9b79-3feb723f2952-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:24.117917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.117912    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krhcm\" (UniqueName: \"kubernetes.io/projected/07bbabd0-544f-4124-9b79-3feb723f2952-kube-api-access-krhcm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:24.481999 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.481947    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a5da41-4722-4572-aa97-b52dbc9c74c3" path="/var/lib/kubelet/pods/b8a5da41-4722-4572-aa97-b52dbc9c74c3/volumes"
Nov 04 01:19:24.482568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.482546    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d480f38d-eb4a-40ed-93ee-31098585bdb4" path="/var/lib/kubelet/pods/d480f38d-eb4a-40ed-93ee-31098585bdb4/volumes"
Nov 04 01:19:24.691075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.691046    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"
Nov 04 01:19:24.691253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.691045    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947" event={"ID":"07bbabd0-544f-4124-9b79-3feb723f2952","Type":"ContainerDied","Data":"5f74c83020e68840f2ee73c78443b099dd98a35a79b9f898a35c3730be4e77ad"}
Nov 04 01:19:24.691253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.691175    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f74c83020e68840f2ee73c78443b099dd98a35a79b9f898a35c3730be4e77ad"
Nov 04 01:19:24.783075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.780609    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:24.986303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:24.986248    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:25.784177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:25.784132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:26.252658 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:19:26.252617    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:19:26.660847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:26.660811    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:27.561597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:27.557586    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:28.307847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:28.307814    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:29.002812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:29.002283    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:29.816468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:29.816367    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:30.141966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:30.141926    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:30.369012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:30.367637    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:30.692637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:30.692607    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:31.024313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:31.024227    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"]
Nov 04 01:19:31.030223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:31.030195    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa80e14-6xvt8"]
Nov 04 01:19:31.270888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:31.270841    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:31.482656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:31.482617    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:31.701718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:31.701681    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:32.009852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.009810    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:32.280772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.280612    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:32.422727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.422692    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"]
Nov 04 01:19:32.424062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.423981    5797 cpu_manager.go:414] "containerMap: removing container" podUID="07bbabd0-544f-4124-9b79-3feb723f2952" containerName="revoke-keys"
Nov 04 01:19:32.424062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.424063    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bbabd0-544f-4124-9b79-3feb723f2952" containerName="revoke-keys"
Nov 04 01:19:32.424266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.424252    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bbabd0-544f-4124-9b79-3feb723f2952" containerName="revoke-keys"
Nov 04 01:19:32.430732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.430705    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.436431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.436403    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"]
Nov 04 01:19:32.469507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.469468    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-3bf4f9-openshift-default-dockercfg-2ndrx\""
Nov 04 01:19:32.494074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.493984    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80281262-60a2-416f-a325-5ffdbe9a4cf1" path="/var/lib/kubelet/pods/80281262-60a2-416f-a325-5ffdbe9a4cf1/volumes"
Nov 04 01:19:32.543790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.543678    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:32.602017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.592017    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.634415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.634370    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.634733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.634709    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.634891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.634875    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.639011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.635074    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4lx\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.639011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.635249    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.639011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.635317    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.639011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.635372    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.639011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.635565    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742110    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742229    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742273    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742300    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4lx\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742414    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742468    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742512    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742640    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.743019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.742688    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.757081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.744248    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.757081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.744574    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.766066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.765457    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.767542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.767517    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.768255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.768230    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.794892 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.794479    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:32.828014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.815549    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.854649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.854579    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.907662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.890544    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4lx\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:32.907662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:32.890754    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy\") pod \"e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") " pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:33.050059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.049966    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:33.280881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.280841    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:33.396156 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:19:33.396124    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d933d6_8278_432e_bdf4_7bbd5d63f8b7.slice/crio-46536e0364602e900bd29c12890e4d6da92984d51a7492eef5cb8e1448c6abd8 WatchSource:0}: Error finding container 46536e0364602e900bd29c12890e4d6da92984d51a7492eef5cb8e1448c6abd8: Status 404 returned error can't find the container with id 46536e0364602e900bd29c12890e4d6da92984d51a7492eef5cb8e1448c6abd8
Nov 04 01:19:33.398782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.398753    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"]
Nov 04 01:19:33.400507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.400466    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:19:33.400595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.400575    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:19:33.400658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.400643    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:19:33.888021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.886979    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" event={"ID":"94d933d6-8278-432e-bdf4-7bbd5d63f8b7","Type":"ContainerStarted","Data":"8ee2c20f41f18de169acce7a515f4bf24c8d59cf0d340f9b941c11f38fa45e21"}
Nov 04 01:19:33.888021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.887079    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" event={"ID":"94d933d6-8278-432e-bdf4-7bbd5d63f8b7","Type":"ContainerStarted","Data":"46536e0364602e900bd29c12890e4d6da92984d51a7492eef5cb8e1448c6abd8"}
Nov 04 01:19:33.943016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:33.937552    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podStartSLOduration=1.93752704 podStartE2EDuration="1.93752704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:33.924308853 +0000 UTC m=+961.566215103" watchObservedRunningTime="2026-11-04 01:19:33.93752704 +0000 UTC m=+961.579433291"
Nov 04 01:19:34.054471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:34.052072    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:34.077011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:34.068763    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:34.382025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:34.381256    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:34.898540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:34.898503    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:34.899508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:34.899478    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:19:35.363784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.363750    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:35.573373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.573323    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:35.644253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.644176    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"]
Nov 04 01:19:35.648729 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.648708    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.653432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.653403    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:19:35.654200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.654159    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:19:35.661119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.661082    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"]
Nov 04 01:19:35.707055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.706968    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.707300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.707260    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.707474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.707437    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swtz\" (UniqueName: \"kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.808796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.808760    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.809010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.808890    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.809101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.809016    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swtz\" (UniqueName: \"kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.809610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.809587    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.811431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.811404    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.821535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.821509    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swtz\" (UniqueName: \"kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz\") pod \"maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:35.969624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:35.969535    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:36.033393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.033359    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"]
Nov 04 01:19:36.044211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.043558    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"]
Nov 04 01:19:36.051467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.050540    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af3eca9-9b7bx"]
Nov 04 01:19:36.059060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.056597    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99973c09-47txl"]
Nov 04 01:19:36.258766 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:19:36.258735    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1122c02c_8b57_49f8_b72c_3994b8dc7ad1.slice/crio-7fb91a7f935c098c4abd6a10a83f887d0e3fe96ad95585a45802c3da54ad10f2 WatchSource:0}: Error finding container 7fb91a7f935c098c4abd6a10a83f887d0e3fe96ad95585a45802c3da54ad10f2: Status 404 returned error can't find the container with id 7fb91a7f935c098c4abd6a10a83f887d0e3fe96ad95585a45802c3da54ad10f2
Nov 04 01:19:36.260188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.260161    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"]
Nov 04 01:19:36.469842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.469808    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:36.482113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.482082    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0abda11a-e17f-421d-85fe-c40fda1026cc" path="/var/lib/kubelet/pods/0abda11a-e17f-421d-85fe-c40fda1026cc/volumes"
Nov 04 01:19:36.482770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.482751    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15154b60-bc67-4d99-b399-ae2dbb675f2c" path="/var/lib/kubelet/pods/15154b60-bc67-4d99-b399-ae2dbb675f2c/volumes"
Nov 04 01:19:36.697311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.697275    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:36.918213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.918114    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw" event={"ID":"1122c02c-8b57-49f8-b72c-3994b8dc7ad1","Type":"ContainerStarted","Data":"1812529085bdee847df2d9e97997322d0fcbdbacfbf630aba8bb1ad9c5095eaa"}
Nov 04 01:19:36.918213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.918186    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw" event={"ID":"1122c02c-8b57-49f8-b72c-3994b8dc7ad1","Type":"ContainerStarted","Data":"7fb91a7f935c098c4abd6a10a83f887d0e3fe96ad95585a45802c3da54ad10f2"}
Nov 04 01:19:36.922518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.922476    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:36.940257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:36.940208    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw" podStartSLOduration=1.940195416 podStartE2EDuration="1.940195416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:36.936322201 +0000 UTC m=+964.578228470" watchObservedRunningTime="2026-11-04 01:19:36.940195416 +0000 UTC m=+964.582101689"
Nov 04 01:19:37.287649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:37.287609    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:37.861979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:37.861944    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:37.937363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:37.937327    5797 generic.go:356] "Generic (PLEG): container finished" podID="1122c02c-8b57-49f8-b72c-3994b8dc7ad1" containerID="1812529085bdee847df2d9e97997322d0fcbdbacfbf630aba8bb1ad9c5095eaa" exitCode=0
Nov 04 01:19:37.937564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:37.937479    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw" event={"ID":"1122c02c-8b57-49f8-b72c-3994b8dc7ad1","Type":"ContainerDied","Data":"1812529085bdee847df2d9e97997322d0fcbdbacfbf630aba8bb1ad9c5095eaa"}
Nov 04 01:19:38.005321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.005270    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"]
Nov 04 01:19:38.014761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.014692    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cca527b7-pz947"]
Nov 04 01:19:38.034316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.034280    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"]
Nov 04 01:19:38.039873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.039848    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c3b6d48-f7xmq"]
Nov 04 01:19:38.068517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.068483    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:38.303009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.299595    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:38.501371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.497804    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bbabd0-544f-4124-9b79-3feb723f2952" path="/var/lib/kubelet/pods/07bbabd0-544f-4124-9b79-3feb723f2952/volumes"
Nov 04 01:19:38.501371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.499017    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3be2608-3dbe-4276-b598-7e3f21d0e420" path="/var/lib/kubelet/pods/e3be2608-3dbe-4276-b598-7e3f21d0e420/volumes"
Nov 04 01:19:38.551979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:38.551930    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:39.153092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.153061    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:39.221363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.219697    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz\" (UniqueName: \"kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz\") pod \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") "
Nov 04 01:19:39.249241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.240268    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca\") pod \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") "
Nov 04 01:19:39.249241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.240448    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca\") pod \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\" (UID: \"1122c02c-8b57-49f8-b72c-3994b8dc7ad1\") "
Nov 04 01:19:39.249241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.248833    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca" pod "1122c02c-8b57-49f8-b72c-3994b8dc7ad1" (UID: "1122c02c-8b57-49f8-b72c-3994b8dc7ad1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:19:39.251973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.251927    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz" pod "1122c02c-8b57-49f8-b72c-3994b8dc7ad1" (UID: "1122c02c-8b57-49f8-b72c-3994b8dc7ad1"). InnerVolumeSpecName "kube-api-access-4swtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:19:39.255273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.254530    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca" pod "1122c02c-8b57-49f8-b72c-3994b8dc7ad1" (UID: "1122c02c-8b57-49f8-b72c-3994b8dc7ad1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:19:39.265153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.265123    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:39.349804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.349769    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:39.349804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.349801    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:39.350046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.349814    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swtz\" (UniqueName: \"kubernetes.io/projected/1122c02c-8b57-49f8-b72c-3994b8dc7ad1-kube-api-access-4swtz\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:19:39.957342 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.957309    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"
Nov 04 01:19:39.957920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.957350    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw" event={"ID":"1122c02c-8b57-49f8-b72c-3994b8dc7ad1","Type":"ContainerDied","Data":"7fb91a7f935c098c4abd6a10a83f887d0e3fe96ad95585a45802c3da54ad10f2"}
Nov 04 01:19:39.957920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:39.957408    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb91a7f935c098c4abd6a10a83f887d0e3fe96ad95585a45802c3da54ad10f2"
Nov 04 01:19:40.037904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:40.037863    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:40.824041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:40.820111    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:41.619035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:41.618945    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:42.027863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:42.027761    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"]
Nov 04 01:19:42.033629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:42.033603    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d10b23-2zk75"]
Nov 04 01:19:42.079202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:42.079168    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:42.292753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:42.288786    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:42.480456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:42.480424    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd" path="/var/lib/kubelet/pods/62acd0eb-2fa1-4f0e-8a1e-77653b3fd0cd/volumes"
Nov 04 01:19:42.944012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:42.941788    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:43.673653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:43.673620    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:43.864328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:43.864260    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:44.031778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:44.031699    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"]
Nov 04 01:19:44.037857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:44.037591    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102ed2f5-csk9r"]
Nov 04 01:19:44.480038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:44.479978    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be6cc25-4163-4669-935b-9a7b276d1bf0" path="/var/lib/kubelet/pods/0be6cc25-4163-4669-935b-9a7b276d1bf0/volumes"
Nov 04 01:19:44.665660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:44.665622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:45.420050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:45.419955    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:45.936406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:45.936357    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:46.181329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:46.181292    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:46.429119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:46.429090    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:47.362242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:47.362145    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:48.174627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:48.174586    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:48.989973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:48.989933    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:49.672931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:49.672891    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:50.329481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:50.328825    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:51.096138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:51.096103    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:51.746146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:51.746110    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:51.950641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:51.950564    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"]
Nov 04 01:19:51.966503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:51.966421    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ab7c587-49gqw"]
Nov 04 01:19:51.988050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:51.987871    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:52.254812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:52.254764    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:52.463252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:52.463215    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:52.485368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:52.485333    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1122c02c-8b57-49f8-b72c-3994b8dc7ad1" path="/var/lib/kubelet/pods/1122c02c-8b57-49f8-b72c-3994b8dc7ad1/volumes"
Nov 04 01:19:52.993857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:52.993818    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:53.030377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:53.030347    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"]
Nov 04 01:19:53.042237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:53.042202    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf5823d-hm8vn"]
Nov 04 01:19:53.217083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:53.217042    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:53.753012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:53.751572    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:54.180627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:54.180590    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:54.386150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:54.386119    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:54.480658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:54.480575    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c45138-01f4-4226-a249-0b786061a5ac" path="/var/lib/kubelet/pods/52c45138-01f4-4226-a249-0b786061a5ac/volumes"
Nov 04 01:19:54.887512 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:54.887481    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:55.678139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:55.678097    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:56.252815 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:19:56.252778    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:19:56.441004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:56.440488    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:57.165321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:57.165278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:57.372751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:57.372702    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:57.582205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:57.582122    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:58.112534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.112494    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:58.132013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.129009    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"]
Nov 04 01:19:58.136929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.136887    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"]
Nov 04 01:19:58.143120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.143079    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81248b03-kl86r"]
Nov 04 01:19:58.148682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.148649    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eafca183-q4zrz"]
Nov 04 01:19:58.487481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.487444    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd941a2d-1526-44f4-8648-3523597644f5" path="/var/lib/kubelet/pods/cd941a2d-1526-44f4-8648-3523597644f5/volumes"
Nov 04 01:19:58.488239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.488213    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9" path="/var/lib/kubelet/pods/fc5efa76-ad37-4da3-8c5c-9f2e5807f5c9/volumes"
Nov 04 01:19:58.587680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.587624    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:58.846737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:58.846575    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:59.047018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.041077    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"]
Nov 04 01:19:59.050069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.050037    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23db2f58-5vflz"]
Nov 04 01:19:59.123761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.123725    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:59.351744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.351709    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"]
Nov 04 01:19:59.352397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.352383    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1122c02c-8b57-49f8-b72c-3994b8dc7ad1" containerName="revoke-keys"
Nov 04 01:19:59.352448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.352398    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1122c02c-8b57-49f8-b72c-3994b8dc7ad1" containerName="revoke-keys"
Nov 04 01:19:59.352534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.352524    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1122c02c-8b57-49f8-b72c-3994b8dc7ad1" containerName="revoke-keys"
Nov 04 01:19:59.357900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.357876    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.362097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.362064    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:19:59.363373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.363354    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:19:59.366335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.366310    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"]
Nov 04 01:19:59.456144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.456054    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.456312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.456243    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.456432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.456418    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm6x\" (UniqueName: \"kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.557802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.557768    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.558340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.557952    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.558340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.558156    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm6x\" (UniqueName: \"kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.558743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.558721    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.560412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.560389    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.572688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.572660    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm6x\" (UniqueName: \"kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x\") pod \"maas-api-revoke-sub-a9707a4e-pzjb7\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.679055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.678982    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:19:59.894657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.894621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:19:59.972536 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:19:59.972490    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f5f427_d647_461a_85ca_f1ca684cac4a.slice/crio-b862817b6ab148bf1d088f4d3113cfc77fc9b2c874892f3a2c5c245cd5da4abd WatchSource:0}: Error finding container b862817b6ab148bf1d088f4d3113cfc77fc9b2c874892f3a2c5c245cd5da4abd: Status 404 returned error can't find the container with id b862817b6ab148bf1d088f4d3113cfc77fc9b2c874892f3a2c5c245cd5da4abd
Nov 04 01:19:59.974198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:19:59.974163    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"]
Nov 04 01:20:00.348765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:00.348715    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7" event={"ID":"22f5f427-d647-461a-85ca-f1ca684cac4a","Type":"ContainerStarted","Data":"b862817b6ab148bf1d088f4d3113cfc77fc9b2c874892f3a2c5c245cd5da4abd"}
Nov 04 01:20:00.480741 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:00.480709    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae8f72b-2344-412e-86b4-a7eb1080569d" path="/var/lib/kubelet/pods/5ae8f72b-2344-412e-86b4-a7eb1080569d/volumes"
Nov 04 01:20:00.498711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:00.498673    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:00.771467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:00.771438    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:01.362681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:01.362640    5797 generic.go:356] "Generic (PLEG): container finished" podID="22f5f427-d647-461a-85ca-f1ca684cac4a" containerID="5caac918a86fee74bd10089ac0415eecfad4148d576e966496ce4fdf253b79e1" exitCode=0
Nov 04 01:20:01.362885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:01.362722    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7" event={"ID":"22f5f427-d647-461a-85ca-f1ca684cac4a","Type":"ContainerDied","Data":"5caac918a86fee74bd10089ac0415eecfad4148d576e966496ce4fdf253b79e1"}
Nov 04 01:20:01.745265 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:01.738981    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:02.027150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.027054    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"]
Nov 04 01:20:02.032036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.032010    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad1e043-hllkj"]
Nov 04 01:20:02.487533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.487499    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c917cb96-fd72-4b11-9c10-2acad6284171" path="/var/lib/kubelet/pods/c917cb96-fd72-4b11-9c10-2acad6284171/volumes"
Nov 04 01:20:02.624138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.621957    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:02.627780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.627712    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:20:02.724154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.724113    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca\") pod \"22f5f427-d647-461a-85ca-f1ca684cac4a\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") "
Nov 04 01:20:02.724372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.724175    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca\") pod \"22f5f427-d647-461a-85ca-f1ca684cac4a\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") "
Nov 04 01:20:02.724372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.724260    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x\" (UniqueName: \"kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x\") pod \"22f5f427-d647-461a-85ca-f1ca684cac4a\" (UID: \"22f5f427-d647-461a-85ca-f1ca684cac4a\") "
Nov 04 01:20:02.724970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.724941    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca" pod "22f5f427-d647-461a-85ca-f1ca684cac4a" (UID: "22f5f427-d647-461a-85ca-f1ca684cac4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:20:02.725350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.725326    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:20:02.726797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.726763    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca" pod "22f5f427-d647-461a-85ca-f1ca684cac4a" (UID: "22f5f427-d647-461a-85ca-f1ca684cac4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:20:02.727382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.727355    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x" pod "22f5f427-d647-461a-85ca-f1ca684cac4a" (UID: "22f5f427-d647-461a-85ca-f1ca684cac4a"). InnerVolumeSpecName "kube-api-access-kpm6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:20:02.827258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.827228    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f5f427-d647-461a-85ca-f1ca684cac4a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:20:02.827258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:02.827256    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpm6x\" (UniqueName: \"kubernetes.io/projected/22f5f427-d647-461a-85ca-f1ca684cac4a-kube-api-access-kpm6x\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:20:03.388261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:03.388223    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:03.403875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:03.403839    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7" event={"ID":"22f5f427-d647-461a-85ca-f1ca684cac4a","Type":"ContainerDied","Data":"b862817b6ab148bf1d088f4d3113cfc77fc9b2c874892f3a2c5c245cd5da4abd"}
Nov 04 01:20:03.404166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:03.404141    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b862817b6ab148bf1d088f4d3113cfc77fc9b2c874892f3a2c5c245cd5da4abd"
Nov 04 01:20:03.404166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:03.404070    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"
Nov 04 01:20:04.095065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:04.095026    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:04.781069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:04.781018    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:05.123902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:05.123863    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:05.326625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:05.326568    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:05.871053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:05.866370    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:06.609086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:06.609045    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:07.148076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:07.148038    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:07.628010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:07.621087    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:08.798014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:08.793951    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:09.103172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.103088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:09.415563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.415529    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"]
Nov 04 01:20:09.416898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.416865    5797 cpu_manager.go:414] "containerMap: removing container" podUID="22f5f427-d647-461a-85ca-f1ca684cac4a" containerName="revoke-keys"
Nov 04 01:20:09.417078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.417062    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f5f427-d647-461a-85ca-f1ca684cac4a" containerName="revoke-keys"
Nov 04 01:20:09.417407 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.417388    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f5f427-d647-461a-85ca-f1ca684cac4a" containerName="revoke-keys"
Nov 04 01:20:09.501609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.501156    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.505047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.502533    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"]
Nov 04 01:20:09.510551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.509670    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e7742169-openshift-default-dockercfg-dp2rm\""
Nov 04 01:20:09.578321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.578224    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:09.641536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.641487    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st82s\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.642725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.642698    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.643207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.643186    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.643510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.643491    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.668331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.668139    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.669409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.669383    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.670371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.670340    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.680592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.680550    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.681405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.681208    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794100    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794179    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794243    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794304    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794393    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794453    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794611    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st82s\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794667    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.794700    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.796252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.795569    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.797916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.797274    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.798366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.798342    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.799096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.799075    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.799455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.799436    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.815564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.815532    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st82s\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.828285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.827605    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:09.840322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.839426    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.848213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.847872    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:09.871334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:09.864839    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-dgng6\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:10.034158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.032752    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:10.057086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.057049    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"]
Nov 04 01:20:10.123041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.116950    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f012d617-lb7xh"]
Nov 04 01:20:10.135570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.135540    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:10.249026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.247570    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:10.506897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.506396    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42d7cdd-3302-40d0-b5b4-98187912f57a" path="/var/lib/kubelet/pods/f42d7cdd-3302-40d0-b5b4-98187912f57a/volumes"
Nov 04 01:20:10.582060 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:20:10.582023    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59912ca4_fdd9_4f5b_9713_220f3136e515.slice/crio-7252c2eff7875708b7102093fc77c9322f4208331ea962e382c21dc5e474d6b7 WatchSource:0}: Error finding container 7252c2eff7875708b7102093fc77c9322f4208331ea962e382c21dc5e474d6b7: Status 404 returned error can't find the container with id 7252c2eff7875708b7102093fc77c9322f4208331ea962e382c21dc5e474d6b7
Nov 04 01:20:10.584541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.584503    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:10.584654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.584614    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:10.584714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.584661    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:10.588160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:10.588135    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"]
Nov 04 01:20:11.032331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.032300    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"]
Nov 04 01:20:11.084011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.076981    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a778bb3e-6tpsq"]
Nov 04 01:20:11.107009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.106533    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:11.389805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.389766    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:11.586810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.586776    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" event={"ID":"59912ca4-fdd9-4f5b-9713-220f3136e515","Type":"ContainerStarted","Data":"8d397e7ba01a8c7e6249629a6961e160a47952c7035421e5823cb0720427a086"}
Nov 04 01:20:11.587036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.586833    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" event={"ID":"59912ca4-fdd9-4f5b-9713-220f3136e515","Type":"ContainerStarted","Data":"7252c2eff7875708b7102093fc77c9322f4208331ea962e382c21dc5e474d6b7"}
Nov 04 01:20:11.618524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:11.618466    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podStartSLOduration=2.618452093 podStartE2EDuration="2.618452093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:11.612380904 +0000 UTC m=+999.254287211" watchObservedRunningTime="2026-11-04 01:20:11.618452093 +0000 UTC m=+999.260358418"
Nov 04 01:20:12.141468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:12.141387    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:12.171161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:12.169809    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:12.172299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:12.172265    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:12.488751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:12.488658    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556cf0e8-60c6-4bb4-b02a-e26de0b56c32" path="/var/lib/kubelet/pods/556cf0e8-60c6-4bb4-b02a-e26de0b56c32/volumes"
Nov 04 01:20:12.622577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:12.622542    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:12.623710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:12.623685    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:13.298345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:13.298295    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:14.079871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:14.079832    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:14.313091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:14.313058    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:15.341132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:15.341008    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:15.600292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:15.593111    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:16.361909 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:16.361872    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:17.189014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:17.185413    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:17.515274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:17.515089    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:18.406667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:18.405562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:19.132740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:19.132684    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:19.863580 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:19.863546    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:20.546787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:20.546755    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:20.941140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:20.941102    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:20:20.941346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:20.941167    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:21.194874 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.194713    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:21.771105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.769963    5797 generic.go:356] "Generic (PLEG): container finished" podID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerID="8ee2c20f41f18de169acce7a515f4bf24c8d59cf0d340f9b941c11f38fa45e21" exitCode=0
Nov 04 01:20:21.771105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.770203    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" event={"ID":"94d933d6-8278-432e-bdf4-7bbd5d63f8b7","Type":"ContainerDied","Data":"8ee2c20f41f18de169acce7a515f4bf24c8d59cf0d340f9b941c11f38fa45e21"}
Nov 04 01:20:21.771105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.770687    5797 scope.go:122] "RemoveContainer" containerID="8ee2c20f41f18de169acce7a515f4bf24c8d59cf0d340f9b941c11f38fa45e21"
Nov 04 01:20:21.772474 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.772441    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:21.772580 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.772516    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:21.772580 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:21.772566    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:22.017931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:22.015550    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:22.735026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:22.734958    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:22.864040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:22.863975    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" event={"ID":"94d933d6-8278-432e-bdf4-7bbd5d63f8b7","Type":"ContainerStarted","Data":"223684cc34028beb4d97c39d2caf26e1029ae0456a40a7c9dd418a4ccc6af764"}
Nov 04 01:20:22.974016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:22.948255    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:23.030347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.030248    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"]
Nov 04 01:20:23.036115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.036080    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"]
Nov 04 01:20:23.041671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.041639    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8131eae8-jcdfr"]
Nov 04 01:20:23.046458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.046431    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849450a3-qvv6p"]
Nov 04 01:20:23.072277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.072190    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:20:23.072884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.072857    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 04 01:20:23.073186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.073148    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 04 01:20:23.284464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.284358    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:23.727168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.727110    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:23.937650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:23.937580    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:24.071073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:24.067156    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 04 01:20:24.071073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:24.067222    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 04 01:20:24.487706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:24.487659    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc7f129-bf83-42a9-b6f3-bb497180eed0" path="/var/lib/kubelet/pods/6dc7f129-bf83-42a9-b6f3-bb497180eed0/volumes"
Nov 04 01:20:24.488396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:24.488372    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb3ce1e-3f6b-4020-a280-845c70a362da" path="/var/lib/kubelet/pods/deb3ce1e-3f6b-4020-a280-845c70a362da/volumes"
Nov 04 01:20:24.892948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:24.892911    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:25.058021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:25.056090    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 04 01:20:25.058021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:25.056161    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 04 01:20:25.822078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:25.822038    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:26.063397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:26.063361    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 04 01:20:26.067319 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:26.067284    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 04 01:20:26.296301 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:20:26.296262    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:20:26.685692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:26.685652    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:27.042618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:27.042532    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"]
Nov 04 01:20:27.071015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:27.054432    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 04 01:20:27.071015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:27.054496    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 04 01:20:27.071015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:27.059852    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c620916-p8c4s"]
Nov 04 01:20:27.767462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:27.767427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:27.973003 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:27.972954    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:28.059085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:28.054398    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 04 01:20:28.059085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:28.054463    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 04 01:20:28.482118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:28.482078    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e1decf-a43d-4327-9936-c9f34af0e317" path="/var/lib/kubelet/pods/14e1decf-a43d-4327-9936-c9f34af0e317/volumes"
Nov 04 01:20:28.973013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:28.972070    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:29.059858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:29.059826    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:20:29.061784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:29.061663    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:20:29.067044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:29.065765    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:20:29.181765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:29.181608    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:30.088112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:30.088072    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:30.287794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:30.287733    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:30.806811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:30.806777    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:31.650656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:31.650621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:31.846959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:31.846927    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:32.385317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:32.385270    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:33.095108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:33.095073    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:33.757114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:33.757077    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:34.665466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:34.665427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:35.409421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:35.409383    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:36.089610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.089573    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:36.173572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.173534    5797 scope.go:122] "RemoveContainer" containerID="5411ea46d65c3dbd73b45741fa33b5db70096ae5689c8ad3f3e1793e73e99fd1"
Nov 04 01:20:36.186130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.186109    5797 scope.go:122] "RemoveContainer" containerID="9628771fbf7f97c9e80fa735ac6d45f107f26c270999b3a409ae0f7bae28ed54"
Nov 04 01:20:36.196985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.196961    5797 scope.go:122] "RemoveContainer" containerID="a0045628e60594bfec193dbc679c4cae2573c977a30557b67f45f94499c90adc"
Nov 04 01:20:36.207809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.207784    5797 scope.go:122] "RemoveContainer" containerID="1b81a980cc46e82ed753ea123e152eef19167345bea8bb3a72099f82fd915e20"
Nov 04 01:20:36.218293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.218270    5797 scope.go:122] "RemoveContainer" containerID="548ed26f398bb1ce58a7c81ac55b0fea409793d97f99fd680c5b5cbd35a72acf"
Nov 04 01:20:36.228605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.228584    5797 scope.go:122] "RemoveContainer" containerID="c5e214f38e7e5525b60c5cd326520d0ea3677ef0a85af8b51bfa79fd6212b699"
Nov 04 01:20:36.960284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:36.960251    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:37.083256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:37.083221    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"]
Nov 04 01:20:37.123766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:37.123722    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aa44a09-fzrnt"]
Nov 04 01:20:37.181878 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:37.181792    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:37.831974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:37.831935    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:38.480944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:38.480913    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0108cf1d-d038-4d3c-888a-fd49a73c8d1c" path="/var/lib/kubelet/pods/0108cf1d-d038-4d3c-888a-fd49a73c8d1c/volumes"
Nov 04 01:20:38.709724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:38.709680    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:39.327553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:39.327513    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:39.527524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:39.527485    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:40.432777 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:40.426749    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:41.243013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:41.240269    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:42.101770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:42.101726    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:42.320587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:42.320132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:43.102895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:43.102857    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:44.015570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:44.015529    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:44.997015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:44.975267    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:45.086013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.063100    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:45.086013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.063979    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:20:45.493152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.489389    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:20:45.530138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.529787    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:45.552144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.552090    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:45.601422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.601366    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:45.752125 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.725662    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:45.769242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.769188    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:45.960222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:45.959135    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:46.041079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.040186    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:20:46.041480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.041425    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:20:46.210016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.192493    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:20:46.236076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.219375    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:20:46.563455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.563382    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:46.615141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.615098    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:20:46.753016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.741920    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:20:46.753016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.742083    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:20:46.971041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.963678    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:46.971041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:46.964081    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:20:47.058292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.041476    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:47.081223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.074314    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:47.195942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.194050    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:20:47.471964 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.468021    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:47.484592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.484550    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:47.491521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.491494    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:20:47.578427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.576361    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:20:47.592488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.590644    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:20:47.615118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.610308    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:47.634969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.634941    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:20:47.696749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.696706    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:20:47.837067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:47.834406    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:20:48.527608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:48.519924    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:49.300087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:49.299468    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:50.112627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:50.112151    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:51.064558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:51.064513    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:51.791866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:51.791759    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:51.999611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:51.999565    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:52.590419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:52.590304    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:53.164300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:53.164267    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:53.461845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:53.461203    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:54.175954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.175833    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:54.503570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.503472    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"]
Nov 04 01:20:54.551696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.551655    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"]
Nov 04 01:20:54.551852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.551805    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698384    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698607    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698714    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698750    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698850    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698907    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698928    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.698973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.698960    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.699141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.699119    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tl2\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801209    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801261    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801312    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801338    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801356    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801497    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801546    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tl2\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801747 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801642    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801787    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801821    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.801929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.801865    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.802131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.802110    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.802249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.802232    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.802320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.802255    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.803866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.803845    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.803981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.803962    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.813721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.813693    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.813887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.813869    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tl2\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2\") pod \"e2e-mt-e7742169-openshift-default-6986f46949-n8jpf\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") " pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.884790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.884442    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:54.911066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:54.909976    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:55.288133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.288096    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"]
Nov 04 01:20:55.290130 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:20:55.290099    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4586618_b0f4_4315_ad8b_f4fdeddbb5f5.slice/crio-39535a322bb806810e56896bd9c450dbd6cc5d6129f38bca06bcc1ac711385e4 WatchSource:0}: Error finding container 39535a322bb806810e56896bd9c450dbd6cc5d6129f38bca06bcc1ac711385e4: Status 404 returned error can't find the container with id 39535a322bb806810e56896bd9c450dbd6cc5d6129f38bca06bcc1ac711385e4
Nov 04 01:20:55.293020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.292970    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:55.293107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.293066    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:55.293107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.293098    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:55.689165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.689117    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:55.923318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.923278    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" event={"ID":"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5","Type":"ContainerStarted","Data":"326ec39ad4f4f95216d4b4903203e268f665c58fc8f405d46d7480cfc8ded9c5"}
Nov 04 01:20:55.923501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:55.923342    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" event={"ID":"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5","Type":"ContainerStarted","Data":"39535a322bb806810e56896bd9c450dbd6cc5d6129f38bca06bcc1ac711385e4"}
Nov 04 01:20:56.258045 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:20:56.257983    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:20:56.385021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.380280    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:56.885691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.885657    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:20:56.887809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.887780    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:20:56.887926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.887842    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:20:56.934724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.934647    5797 generic.go:356] "Generic (PLEG): container finished" podID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerID="8d397e7ba01a8c7e6249629a6961e160a47952c7035421e5823cb0720427a086" exitCode=0
Nov 04 01:20:56.935247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.934720    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" event={"ID":"59912ca4-fdd9-4f5b-9713-220f3136e515","Type":"ContainerDied","Data":"8d397e7ba01a8c7e6249629a6961e160a47952c7035421e5823cb0720427a086"}
Nov 04 01:20:56.935327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.935253    5797 scope.go:122] "RemoveContainer" containerID="8d397e7ba01a8c7e6249629a6961e160a47952c7035421e5823cb0720427a086"
Nov 04 01:20:56.936889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.936861    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:56.937021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.936925    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:56.937021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.936959    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:20:56.998205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:56.997093    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podStartSLOduration=2.997077186 podStartE2EDuration="2.997077186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:55.948292748 +0000 UTC m=+1043.590198999" watchObservedRunningTime="2026-11-04 01:20:56.997077186 +0000 UTC m=+1044.638983438"
Nov 04 01:20:57.199897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:57.199870    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:57.622896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:57.622826    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:57.886066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:57.885957    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:20:57.886066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:57.886033    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:20:57.994011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:57.991667    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" event={"ID":"59912ca4-fdd9-4f5b-9713-220f3136e515","Type":"ContainerStarted","Data":"d5bcb61c5b5a47b85c4de69d1787f30f8cbc7c48ba4cf467bd9b2c3ed1897d94"}
Nov 04 01:20:58.013012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.009284    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:58.142636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.138002    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:20:58.142636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.138924    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 04 01:20:58.142636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.138978    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 04 01:20:58.808060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.807429    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:58.900015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.896476    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:20:58.900015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:58.896555    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:20:59.136593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:59.136560    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 04 01:20:59.136764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:59.136616    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 04 01:20:59.674013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:59.672607    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:20:59.893015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:59.892401    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:20:59.893015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:20:59.892471    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:21:00.136676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:00.136641    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 04 01:21:00.136850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:00.136700    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 04 01:21:00.710021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:00.709964    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:00.887442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:00.887405    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:21:00.887710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:00.887677    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:21:01.137141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:01.137099    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 04 01:21:01.137629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:01.137599    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 04 01:21:01.623176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:01.610594    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:01.902329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:01.902227    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:21:01.902329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:01.902312    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:21:02.136310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:02.136275    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 04 01:21:02.136514 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:02.136347    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 04 01:21:02.572014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:02.558758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:02.894085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:02.894056    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:21:02.895433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:02.895396    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:21:02.897411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:02.897395    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:21:03.137212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:03.136879    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 04 01:21:03.137212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:03.136929    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 04 01:21:03.407407 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:03.407330    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:04.229022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:04.226083    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:21:04.229022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:04.228047    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:21:04.234902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:04.234875    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:21:04.548262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:04.548124    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:05.033344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:05.033215    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"]
Nov 04 01:21:05.044894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:05.042471    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61f985e-l6l8b"]
Nov 04 01:21:05.449323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:05.449219    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:06.321312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.321276    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:06.486064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.486029    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44ce06f-4dca-4e98-bfe7-f29944d8dcea" path="/var/lib/kubelet/pods/a44ce06f-4dca-4e98-bfe7-f29944d8dcea/volumes"
Nov 04 01:21:06.538559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.538524    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:06.614822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.614787    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"]
Nov 04 01:21:06.619421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.619397    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.627572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.627525    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0741ef-openshift-default-dockercfg-vvp2h\""
Nov 04 01:21:06.636427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.636400    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"]
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761326    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761413    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761449    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761525    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761587    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761746    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761773    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761819    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.767133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.761855    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kf72\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.800061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.800012    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869042    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869092    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869175    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869234    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869381    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869404    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869451    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869484    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kf72\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.869636    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.870137    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.871118    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.871382    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.875063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.871619    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.892670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.892636    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.913004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.912923    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.921696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.921610    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.930811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.930743    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kf72\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.951012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.950430    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy\") pod \"e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") " pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:06.963012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:06.956519    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:07.235567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.235471    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:07.349546 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:21:07.349509    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f6a452_5b44_4a9c_b3d3_f7508f95fa38.slice/crio-bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac WatchSource:0}: Error finding container bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac: Status 404 returned error can't find the container with id bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac
Nov 04 01:21:07.352645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.352610    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"]
Nov 04 01:21:07.353658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.353622    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:07.353779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.353710    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:07.353847 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.353785    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:07.534179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.534088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:07.738320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:07.738273    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:08.189218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:08.189178    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" event={"ID":"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38","Type":"ContainerStarted","Data":"38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"}
Nov 04 01:21:08.189521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:08.189234    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" event={"ID":"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38","Type":"ContainerStarted","Data":"bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac"}
Nov 04 01:21:08.216738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:08.216682    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podStartSLOduration=2.216669113 podStartE2EDuration="2.216669113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:08.212457822 +0000 UTC m=+1055.854364139" watchObservedRunningTime="2026-11-04 01:21:08.216669113 +0000 UTC m=+1055.858575362"
Nov 04 01:21:08.677423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:08.677387    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:08.953077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:08.949958    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:08.975778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:08.972128    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:09.029624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:09.029587    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:09.178608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:09.178573    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:09.198826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:09.198796    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:09.199868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:09.199848    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:21:09.672185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:09.672115    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:10.309052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.309011    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"]
Nov 04 01:21:10.314070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.314044    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.321315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.321287    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"]
Nov 04 01:21:10.346751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.346718    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:21:10.346904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.346766    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:21:10.366535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.366499    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.366688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.366631    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.366742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.366718    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8jw\" (UniqueName: \"kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.468719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.468683    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.468969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.468737    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.468969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.468768    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8jw\" (UniqueName: \"kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.469443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.469416    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.471246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.471221    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.480878 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.480857    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8jw\" (UniqueName: \"kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw\") pod \"maas-api-revoke-sub-3504be9a-5wddh\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.576339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.576222    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:10.643636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.643596    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:10.881354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.881322    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:10.935966 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:21:10.935930    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8a7b7a_fdf6_41d6_8406_9debf8a531a2.slice/crio-2d360667d52b2517a1e058ca970f010f58620a8042935b4ae95debb1811aa92a WatchSource:0}: Error finding container 2d360667d52b2517a1e058ca970f010f58620a8042935b4ae95debb1811aa92a: Status 404 returned error can't find the container with id 2d360667d52b2517a1e058ca970f010f58620a8042935b4ae95debb1811aa92a
Nov 04 01:21:10.937973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:10.937942    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"]
Nov 04 01:21:11.083510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:11.083478    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:11.218439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:11.218408    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh" event={"ID":"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2","Type":"ContainerStarted","Data":"2d360667d52b2517a1e058ca970f010f58620a8042935b4ae95debb1811aa92a"}
Nov 04 01:21:11.600894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:11.600779    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:12.033941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.033908    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"]
Nov 04 01:21:12.039394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.039369    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1cae41-5cdrw"]
Nov 04 01:21:12.094358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.094321    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:12.349069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.320818    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:12.558780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.558735    5797 generic.go:356] "Generic (PLEG): container finished" podID="ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" containerID="d3c63f2aa922549a146c2b064dcab074ddd97d6b1dfde8db1f17fcf9cb7c38a5" exitCode=0
Nov 04 01:21:12.566092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.566055    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb14c7b9-b8d3-41f4-aac8-9e79d27472b3" path="/var/lib/kubelet/pods/eb14c7b9-b8d3-41f4-aac8-9e79d27472b3/volumes"
Nov 04 01:21:12.566643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.566623    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh" event={"ID":"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2","Type":"ContainerDied","Data":"d3c63f2aa922549a146c2b064dcab074ddd97d6b1dfde8db1f17fcf9cb7c38a5"}
Nov 04 01:21:12.720564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:12.720467    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:13.468444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.468400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:13.749022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.748978    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:13.806404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.806372    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca\") pod \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") "
Nov 04 01:21:13.806404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.806407    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw\" (UniqueName: \"kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw\") pod \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") "
Nov 04 01:21:13.806765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.806742    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca\") pod \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\" (UID: \"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2\") "
Nov 04 01:21:13.807090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.807066    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca" pod "ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" (UID: "ea8a7b7a-fdf6-41d6-8406-9debf8a531a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:21:13.807413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.807395    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:21:13.809562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.809528    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca" pod "ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" (UID: "ea8a7b7a-fdf6-41d6-8406-9debf8a531a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:21:13.809672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.809578    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw" pod "ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" (UID: "ea8a7b7a-fdf6-41d6-8406-9debf8a531a2"). InnerVolumeSpecName "kube-api-access-nf8jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:21:13.908870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.908833    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:21:13.908870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:13.908864    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8jw\" (UniqueName: \"kubernetes.io/projected/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2-kube-api-access-nf8jw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:21:14.147227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:14.145475    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:14.579899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:14.579864    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"
Nov 04 01:21:14.580112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:14.579899    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh" event={"ID":"ea8a7b7a-fdf6-41d6-8406-9debf8a531a2","Type":"ContainerDied","Data":"2d360667d52b2517a1e058ca970f010f58620a8042935b4ae95debb1811aa92a"}
Nov 04 01:21:14.580112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:14.579944    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d360667d52b2517a1e058ca970f010f58620a8042935b4ae95debb1811aa92a"
Nov 04 01:21:14.915907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:14.915873    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:15.699800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:15.699765    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:16.364520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:16.364483    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:17.188300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:17.188142    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:17.894018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:17.893305    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:18.600011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:18.594467    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:19.239677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:19.239132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:20.008011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:20.000939    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:20.037915 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:20.037849    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"]
Nov 04 01:21:20.046685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:20.046530    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc7868d-28twq"]
Nov 04 01:21:20.222434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:20.222403    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:20.518894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:20.518847    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ce2c49-d221-41ec-a8a3-c4acaa95cb71" path="/var/lib/kubelet/pods/a7ce2c49-d221-41ec-a8a3-c4acaa95cb71/volumes"
Nov 04 01:21:20.753142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:20.753103    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:21.026118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:21.026075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:21.978507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:21.977591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:22.646747 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:22.646705    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:23.312266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:23.312182    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:23.999011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:23.998669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:24.821532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:24.821466    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:25.908217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:25.908158    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:26.286757 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:21:26.286666    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:21:26.517611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:26.517574    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:21:27.059025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:27.058185    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"]
Nov 04 01:21:27.067021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:27.066241    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"]
Nov 04 01:21:27.074020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:27.073895    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50049503-rgfp2"]
Nov 04 01:21:27.081411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:27.081294    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fcccd6f-vjbml"]
Nov 04 01:21:27.173060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:27.173022    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:27.428636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:27.428593    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:28.609528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:28.609494    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b127ba27-cb9c-472b-ba77-c227b5962076" path="/var/lib/kubelet/pods/b127ba27-cb9c-472b-ba77-c227b5962076/volumes"
Nov 04 01:21:28.632747 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:28.632705    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9be1452-095c-4dfc-9862-49ef80ac8349" path="/var/lib/kubelet/pods/f9be1452-095c-4dfc-9862-49ef80ac8349/volumes"
Nov 04 01:21:28.679739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:28.679632    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:29.076177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:29.076109    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:21:29.217014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:29.216402    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:21:29.499139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:29.499078    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:21:29.679015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:29.678147    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:21:29.721087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:29.720113    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:21:30.078408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:30.078275    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:21:30.078970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:30.078935    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:21:30.402069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:30.367637    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:30.470044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:30.458067    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:21:30.562771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:30.562744    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:21:30.572623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:30.572589    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:31.150150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:31.150084    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:31.690911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:31.690876    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:32.106763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:32.106673    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:32.818017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:32.802927    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:33.538608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:33.538563    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:34.440882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:34.440834    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:34.993051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:34.993014    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"]
Nov 04 01:21:34.993749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:34.993735    5797 cpu_manager.go:414] "containerMap: removing container" podUID="ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" containerName="revoke-keys"
Nov 04 01:21:34.993794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:34.993752    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" containerName="revoke-keys"
Nov 04 01:21:34.993871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:34.993862    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" containerName="revoke-keys"
Nov 04 01:21:34.999929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:34.999900    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.004120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.004090    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:21:35.005187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.005134    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:21:35.005814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.005790    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"]
Nov 04 01:21:35.181010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.178608    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.181010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.178678    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.181010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.178796    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmns5\" (UniqueName: \"kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.202751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.200665    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:35.281136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.281063    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.281136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.281099    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.281375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.281198    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmns5\" (UniqueName: \"kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.281750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.281724    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.283501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.283476    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.296539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.296518    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmns5\" (UniqueName: \"kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5\") pod \"maas-api-revoke-sub-6c2aaa5d-z6nqm\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.318424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.318402    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:35.605022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.604973    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"]
Nov 04 01:21:35.605022 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:21:35.605007    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc39046_127e_40c2_aed6_08fbcaafe393.slice/crio-f8f1162d8f62147399c781a87e9af7aa962d94af729be8eebabf44667d863db4 WatchSource:0}: Error finding container f8f1162d8f62147399c781a87e9af7aa962d94af729be8eebabf44667d863db4: Status 404 returned error can't find the container with id f8f1162d8f62147399c781a87e9af7aa962d94af729be8eebabf44667d863db4
Nov 04 01:21:35.890377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:35.890344    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:36.118866 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.118827    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:36.120264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.120234    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm" event={"ID":"5bc39046-127e-40c2-aed6-08fbcaafe393","Type":"ContainerStarted","Data":"5dcc25ae34d721f699fe693467a9b29bca81dc4a7f3407603fe3337c953e596d"}
Nov 04 01:21:36.120429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.120294    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm" event={"ID":"5bc39046-127e-40c2-aed6-08fbcaafe393","Type":"ContainerStarted","Data":"f8f1162d8f62147399c781a87e9af7aa962d94af729be8eebabf44667d863db4"}
Nov 04 01:21:36.145184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.145027    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm" podStartSLOduration=2.145012222 podStartE2EDuration="2.145012222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:36.138109175 +0000 UTC m=+1083.780015444" watchObservedRunningTime="2026-11-04 01:21:36.145012222 +0000 UTC m=+1083.786918473"
Nov 04 01:21:36.298253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.298222    5797 scope.go:122] "RemoveContainer" containerID="704171a5e92ca97b08a3b41377fd9d6a3825e3dd86cc3b6b25e8d74d8bd4a1c0"
Nov 04 01:21:36.310442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.310419    5797 scope.go:122] "RemoveContainer" containerID="201ac6659cd295f323db3532b0981e3a7ddcb8c711919e029fea1ef57200579d"
Nov 04 01:21:36.320699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.320676    5797 scope.go:122] "RemoveContainer" containerID="3db0849f0a39c61d3603c715701fee3d19a7b1dc45a9d0cccc0cdad5d806d5d4"
Nov 04 01:21:36.335506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.332187    5797 scope.go:122] "RemoveContainer" containerID="132beb9ad93bf941bb8da31ee1d2957bf32addc0d35848ba0029ee4261d0aab4"
Nov 04 01:21:36.346053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.345974    5797 scope.go:122] "RemoveContainer" containerID="f25397224eea6d11180ccf1eacd0d7c0102c8d9630d344b9a4453e95af40ae55"
Nov 04 01:21:36.356280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.356257    5797 scope.go:122] "RemoveContainer" containerID="f8ed229bc98879386cc2aa671b6e3f4335733345e6311d1df3f1d8a0b4b184d9"
Nov 04 01:21:36.366448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.366426    5797 scope.go:122] "RemoveContainer" containerID="94120c38e9174fd3a1619963b4cb0bd9ccbc808ef87dea1127557c55cc779319"
Nov 04 01:21:36.376579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.376557    5797 scope.go:122] "RemoveContainer" containerID="6cb8b0c1ba9f6bf2139b8d76618e109234c847976eb820b3e7a12b652ba8e13b"
Nov 04 01:21:36.386347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.386325    5797 scope.go:122] "RemoveContainer" containerID="810dcdc2f8cdd99241a3d36fc36a5c2dda356fad7eae45a6d4b3a3942baf3efc"
Nov 04 01:21:36.395829 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.395807    5797 scope.go:122] "RemoveContainer" containerID="fbc0a0995a81f88e3d8390fad1b76d0eef44eb174772dcb67f39f0974526af01"
Nov 04 01:21:36.405879 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.405854    5797 scope.go:122] "RemoveContainer" containerID="fac79af8d9e6aa0d71749378b3c9c41a86dff7ea603a2fd66c2545f471a966a6"
Nov 04 01:21:36.415632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.415614    5797 scope.go:122] "RemoveContainer" containerID="cb4b32f76e1d731f3e077e528dab64d20182c3a55aa51d6b92180f8184bbfc34"
Nov 04 01:21:36.425439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.425421    5797 scope.go:122] "RemoveContainer" containerID="ebab855f476643304fd7ebeb558edb7cb0cd7130a8304a11182de0e376184bdf"
Nov 04 01:21:36.633690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.632145    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:36.968579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.968543    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:36.982602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.982568    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"]
Nov 04 01:21:36.988751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.988727    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.000219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:36.998354    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5ce9e8-openshift-default-dockercfg-spgl6\""
Nov 04 01:21:37.004643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.004618    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"]
Nov 04 01:21:37.015355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015321    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015433    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015482    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015513    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57slq\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015551    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015583    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015629    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015843    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.015970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.015957    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.136900    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137130    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137230    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137361    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137427    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137460    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57slq\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137511    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137542    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.137604    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.139209    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.139505    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.139782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.139737    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.175080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.163793    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.176240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.175897    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.183070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.180812    5797 generic.go:356] "Generic (PLEG): container finished" podID="5bc39046-127e-40c2-aed6-08fbcaafe393" containerID="5dcc25ae34d721f699fe693467a9b29bca81dc4a7f3407603fe3337c953e596d" exitCode=0
Nov 04 01:21:37.183070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.181087    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm" event={"ID":"5bc39046-127e-40c2-aed6-08fbcaafe393","Type":"ContainerDied","Data":"5dcc25ae34d721f699fe693467a9b29bca81dc4a7f3407603fe3337c953e596d"}
Nov 04 01:21:37.184171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.184145    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57slq\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.189194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.189171    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.190075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.190053    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.191293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.190604    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token\") pod \"e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") " pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.198755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.198695    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:37.307658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.307577    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:37.622200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.622155    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:37.757648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.757614    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"]
Nov 04 01:21:37.762626 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:21:37.762590    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8a8ff8_1e23_47d0_8c8a_a26e7394ccfe.slice/crio-c732938faae343414fc997ed0ca30cca6769cfd75182b4da2c57558072b29ad1 WatchSource:0}: Error finding container c732938faae343414fc997ed0ca30cca6769cfd75182b4da2c57558072b29ad1: Status 404 returned error can't find the container with id c732938faae343414fc997ed0ca30cca6769cfd75182b4da2c57558072b29ad1
Nov 04 01:21:37.767482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.767451    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:37.767675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.767655    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:37.767779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.767767    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:37.822872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:37.822808    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:38.192300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.192257    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" event={"ID":"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe","Type":"ContainerStarted","Data":"f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"}
Nov 04 01:21:38.192471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.192309    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" event={"ID":"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe","Type":"ContainerStarted","Data":"c732938faae343414fc997ed0ca30cca6769cfd75182b4da2c57558072b29ad1"}
Nov 04 01:21:38.223143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.223092    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podStartSLOduration=2.223071762 podStartE2EDuration="2.223071762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:38.214968183 +0000 UTC m=+1085.856874450" watchObservedRunningTime="2026-11-04 01:21:38.223071762 +0000 UTC m=+1085.864978013"
Nov 04 01:21:38.310328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.310295    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:38.318126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.318101    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:38.508658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.508630    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:38.612394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.612350    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca\") pod \"5bc39046-127e-40c2-aed6-08fbcaafe393\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") "
Nov 04 01:21:38.612582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.612540    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5\" (UniqueName: \"kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5\") pod \"5bc39046-127e-40c2-aed6-08fbcaafe393\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") "
Nov 04 01:21:38.612678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.612661    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca\") pod \"5bc39046-127e-40c2-aed6-08fbcaafe393\" (UID: \"5bc39046-127e-40c2-aed6-08fbcaafe393\") "
Nov 04 01:21:38.612743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.612694    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca" pod "5bc39046-127e-40c2-aed6-08fbcaafe393" (UID: "5bc39046-127e-40c2-aed6-08fbcaafe393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:21:38.613874 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.613841    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:21:38.614875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.614851    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5" pod "5bc39046-127e-40c2-aed6-08fbcaafe393" (UID: "5bc39046-127e-40c2-aed6-08fbcaafe393"). InnerVolumeSpecName "kube-api-access-bmns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:21:38.614984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.614873    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca" pod "5bc39046-127e-40c2-aed6-08fbcaafe393" (UID: "5bc39046-127e-40c2-aed6-08fbcaafe393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:21:38.687056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.682828    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:38.715446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.715355    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmns5\" (UniqueName: \"kubernetes.io/projected/5bc39046-127e-40c2-aed6-08fbcaafe393-kube-api-access-bmns5\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:21:38.715693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:38.715655    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc39046-127e-40c2-aed6-08fbcaafe393-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:21:39.206723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:39.206689    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"
Nov 04 01:21:39.206723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:39.206717    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm" event={"ID":"5bc39046-127e-40c2-aed6-08fbcaafe393","Type":"ContainerDied","Data":"f8f1162d8f62147399c781a87e9af7aa962d94af729be8eebabf44667d863db4"}
Nov 04 01:21:39.207329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:39.206765    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f1162d8f62147399c781a87e9af7aa962d94af729be8eebabf44667d863db4"
Nov 04 01:21:39.207329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:39.207042    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:39.207975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:39.207945    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:21:39.579064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:39.572653    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:40.510108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.510066    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:40.715947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.715873    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:40.932386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.932350    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:40.954058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.954020    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"]
Nov 04 01:21:40.955158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.955133    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5bc39046-127e-40c2-aed6-08fbcaafe393" containerName="revoke-keys"
Nov 04 01:21:40.955305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.955157    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc39046-127e-40c2-aed6-08fbcaafe393" containerName="revoke-keys"
Nov 04 01:21:40.955399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.955384    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc39046-127e-40c2-aed6-08fbcaafe393" containerName="revoke-keys"
Nov 04 01:21:40.959541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.959513    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:40.970652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.969397    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"]
Nov 04 01:21:40.970652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:40.969809    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27bc4f8b-openshift-default-dockercfg-6dxmm\""
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.103859    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.103976    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2kp\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104077    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104136    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104240    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104291    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104325    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104366    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.106013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.104400    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.156849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.155655    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206135    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206275    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206331    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206370    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206413    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206446    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206619    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206726    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2kp\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.206807    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.208076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.207349    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.216493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.216464    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.242679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.220331    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.242679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.222524    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.242679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.235944    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.258542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.255664    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2kp\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.301891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.301390    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.302669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.302641    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.315443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.312436    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert\") pod \"e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") " pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.590257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.590161    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:41.600601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.600536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:41.808952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.808913    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:41.955376 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:21:41.955335    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3056bef_bc61_4c70_8aa9_1a0673bf5733.slice/crio-7caab63d7dc3359ec47c8ecfdb297ce496f71d51da7a355ff8ac9d697c568835 WatchSource:0}: Error finding container 7caab63d7dc3359ec47c8ecfdb297ce496f71d51da7a355ff8ac9d697c568835: Status 404 returned error can't find the container with id 7caab63d7dc3359ec47c8ecfdb297ce496f71d51da7a355ff8ac9d697c568835
Nov 04 01:21:41.956485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.956463    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"]
Nov 04 01:21:41.958313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.958283    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:41.958384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.958364    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:41.958424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:41.958393    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:21:42.156225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:42.156187    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:42.417014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:42.393958    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:42.674589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:42.674480    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" event={"ID":"b3056bef-bc61-4c70-8aa9-1a0673bf5733","Type":"ContainerStarted","Data":"f5a4cf00c879b563379c62f69ed751cdf4158c744c2244d4d1cdc547584215bb"}
Nov 04 01:21:42.674589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:42.674551    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" event={"ID":"b3056bef-bc61-4c70-8aa9-1a0673bf5733","Type":"ContainerStarted","Data":"7caab63d7dc3359ec47c8ecfdb297ce496f71d51da7a355ff8ac9d697c568835"}
Nov 04 01:21:42.702845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:42.702135    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podStartSLOduration=2.7021153350000002 podStartE2EDuration="2.702115335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:42.70110853 +0000 UTC m=+1090.343014781" watchObservedRunningTime="2026-11-04 01:21:42.702115335 +0000 UTC m=+1090.344021586"
Nov 04 01:21:43.028220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.028123    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:43.044060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.044017    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"]
Nov 04 01:21:43.048972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.048949    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037160aa-85fzw"]
Nov 04 01:21:43.590737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.590704    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:43.597160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.597134    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:43.695455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.695424    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:43.697144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.697121    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:21:43.716817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:43.716786    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:44.542013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:44.539404    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134d6eab-526f-4cbe-b939-93ac1104337e" path="/var/lib/kubelet/pods/134d6eab-526f-4cbe-b939-93ac1104337e/volumes"
Nov 04 01:21:44.559013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:44.553086    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:45.230253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:45.230220    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:45.461603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:45.461531    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:46.335957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:46.335920    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:46.701033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:46.700945    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:46.986108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:46.958165    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:47.522446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:47.520754    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:48.029850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:48.029814    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"]
Nov 04 01:21:48.034456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:48.034429    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d864-62bfl"]
Nov 04 01:21:48.308961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:48.308883    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:48.488902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:48.488863    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef2953c-0652-404d-8386-607b0f909068" path="/var/lib/kubelet/pods/2ef2953c-0652-404d-8386-607b0f909068/volumes"
Nov 04 01:21:49.105957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:49.105921    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:49.873823 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:49.872070    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:50.044664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:50.044624    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"]
Nov 04 01:21:50.049760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:50.049735    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcbcdaea-6plvs"]
Nov 04 01:21:50.480932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:50.480898    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7" path="/var/lib/kubelet/pods/cf05aeff-5a1b-4ee9-ae25-ddc8a30815b7/volumes"
Nov 04 01:21:50.586842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:50.586717    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:51.028103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:51.028070    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"]
Nov 04 01:21:51.033811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:51.033782    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdae9ea-jkxkz"]
Nov 04 01:21:51.376930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:51.376893    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:51.619667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:51.619632    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:52.108842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:52.108803    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:52.308065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:52.308035    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:52.620477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:52.620409    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e8cf0c-cdbf-40ea-ba23-05486ceedf82" path="/var/lib/kubelet/pods/d4e8cf0c-cdbf-40ea-ba23-05486ceedf82/volumes"
Nov 04 01:21:53.101069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:53.100673    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:53.101069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:53.100766    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"]
Nov 04 01:21:53.296013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:53.278390    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b8d5271-7p2h2"]
Nov 04 01:21:53.880897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:53.880864    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:54.480690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:54.480656    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f" path="/var/lib/kubelet/pods/1f7b644c-fbc0-436d-8fd1-88ea9b9c3a0f/volumes"
Nov 04 01:21:54.591713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:54.591625    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:55.370755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:55.370711    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:56.073461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:56.073400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:56.255206 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:21:56.255164    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:21:56.808048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:56.807588    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:57.027706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:57.027673    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:57.038810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:57.038782    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"]
Nov 04 01:21:57.043174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:57.043153    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5c865c0-xj9gm"]
Nov 04 01:21:57.709370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:57.709281    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:58.481489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:58.481457    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e1bcf3-7f5e-48bd-8978-558783c36c4a" path="/var/lib/kubelet/pods/72e1bcf3-7f5e-48bd-8978-558783c36c4a/volumes"
Nov 04 01:21:58.668035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:58.667960    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:21:59.502454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:21:59.502412    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:00.170693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:00.170624    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:00.871323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:00.871286    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:01.611736 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:01.611703    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:02.407161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:02.407121    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:03.214358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:03.212842    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:03.889163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:03.889125    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:04.852493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:04.852456    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:06.350925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:06.350882    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:07.857964 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:07.857920    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:08.845142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:08.845092    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:22:09.220629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.220584    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:22:09.529386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.529284    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:09.639517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.639477    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:09.697639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.697601    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:09.728749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.723156    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:09.852066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.846206    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:09.852066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:09.846554    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:22:10.233019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.231447    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:10.233019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.232045    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:22:10.598157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.582118    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:10.656017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.639852    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:10.858660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.858552    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:22:10.934044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.933896    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:10.934044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.934005    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:10.935016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.934971    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:22:10.935163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.935053    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:22:10.984113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.984077    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:10.984384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:10.984356    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:11.446088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.446046    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:11.631016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.630108    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:11.631230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.631145    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:22:11.686069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.665271    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:11.712228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.694136    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:11.718307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.717810    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:11.728585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.728200    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:22:11.903780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:11.903732    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:12.744793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:12.744716    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:12.963531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:12.963492    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:13.900974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:13.900937    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:14.988894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:14.988738    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:15.124242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.124200    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"]
Nov 04 01:22:15.163486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.163450    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"]
Nov 04 01:22:15.171521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.171487    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"]
Nov 04 01:22:15.176409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.176378    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a072f6b-gfbzl"]
Nov 04 01:22:15.181516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.181486    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bc74ea-m5zss"]
Nov 04 01:22:15.188625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.188598    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c390d9-8kwqz"]
Nov 04 01:22:15.794814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:15.794688    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:16.034972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:16.034938    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:16.486343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:16.486304    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440d2009-5efa-45d4-b976-574d70d33054" path="/var/lib/kubelet/pods/440d2009-5efa-45d4-b976-574d70d33054/volumes"
Nov 04 01:22:16.487542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:16.487518    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9e5234-5778-4c6b-8747-cdabefef5d3c" path="/var/lib/kubelet/pods/8d9e5234-5778-4c6b-8747-cdabefef5d3c/volumes"
Nov 04 01:22:16.488810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:16.488785    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1a3431-f7b6-4455-8877-07ce2a668170" path="/var/lib/kubelet/pods/9c1a3431-f7b6-4455-8877-07ce2a668170/volumes"
Nov 04 01:22:16.708509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:16.708475    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:16.921552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:16.921508    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:17.370027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:17.369957    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:18.611053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:18.610937    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:18.624937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:18.624355    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:22:19.368942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:19.368904    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:20.101236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:20.101020    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:21.536374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:21.536333    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"]
Nov 04 01:22:21.557739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:21.557711    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7418de00-qklml"]
Nov 04 01:22:22.024692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.024656    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"]
Nov 04 01:22:22.029017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.028982    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5620c6bb-9875l"]
Nov 04 01:22:22.480877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.480844    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2843f357-e6c6-456c-8e72-965e60ec35f7" path="/var/lib/kubelet/pods/2843f357-e6c6-456c-8e72-965e60ec35f7/volumes"
Nov 04 01:22:22.525323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.481511    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c488997b-e965-4fcf-a24f-a329a790b438" path="/var/lib/kubelet/pods/c488997b-e965-4fcf-a24f-a329a790b438/volumes"
Nov 04 01:22:22.678845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.678799    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": context deadline exceeded" start-of-body=
Nov 04 01:22:22.815171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.678885    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": context deadline exceeded"
Nov 04 01:22:22.912165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.912126    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:22.912335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.912200    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:22.998561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:22.921094    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:23.500136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:23.500091    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:23.650030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:23.649971    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:23.867141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:23.867103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:24.084340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:24.084299    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:24.378145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:24.378094    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:24.492942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:24.492907    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:22:24.860694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:24.860607    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:22:25.038845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038784    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.039411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038874    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.039411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038784    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.039411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038784    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.039411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038784    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.039411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038784    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.039411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.038784    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 04 01:22:25.366491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.241147    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:25.366491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.241214    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:25.565116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.565064    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:25.678589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.565276    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:22:25.678589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.582569    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:22:25.678589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:25.596149    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:22:26.352366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:26.352329    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:26.352975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:26.352932    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:27.041105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.041065    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"]
Nov 04 01:22:27.045845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.045813    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea9851d-hgrg6"]
Nov 04 01:22:27.386176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.386124    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:27.548475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.548430    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:22:27.548650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.548520    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:22:27.969179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.969139    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:27.969370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:27.969209    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:28.040430 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:28.040387    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output=""
Nov 04 01:22:28.115363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:28.040506    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:22:28.115363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:28.067973    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output=""
Nov 04 01:22:28.480641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:28.480609    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1adcbf-a7de-4d0e-a138-2c3b7ce0129b" path="/var/lib/kubelet/pods/da1adcbf-a7de-4d0e-a138-2c3b7ce0129b/volumes"
Nov 04 01:22:29.104601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.104560    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"]
Nov 04 01:22:29.378175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.378127    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:29.378401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.378220    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:29.455213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.455157    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.466709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.466091    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:22:29.489878 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.487375    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:22:29.500953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.500570    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.521145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.501215    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9wbq\" (UniqueName: \"kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.521145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.501629    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.603441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.603391    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.662707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.603531    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.662707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.603582    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9wbq\" (UniqueName: \"kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.662707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:29.604248    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:29.838673 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:22:29.825228    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:22:30.037680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:30.037592    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:30.371158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:30.371099    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:32.411094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:31.037985    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:32.411094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:31.038264    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9wbq\" (UniqueName: \"kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq\") pod \"maas-api-revoke-sub-83e7c225-b796p\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:32.411094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:31.152830    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:32.411094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:31.752614    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:33.528092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:32.419842    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:33.528092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:32.651284    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"]
Nov 04 01:22:33.528092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:33.050229    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"]
Nov 04 01:22:33.528092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:33.056930    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81529f6-8gd8f"]
Nov 04 01:22:33.528092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:33.070737    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:33.784857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:33.784749    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:34.480082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:34.480051    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f8ec37-fd43-4212-97ae-358737c09e33" path="/var/lib/kubelet/pods/20f8ec37-fd43-4212-97ae-358737c09e33/volumes"
Nov 04 01:22:34.550016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:34.549961    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:22:35.915488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:35.915453    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:36.520545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:36.520480    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:36.621364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:36.621332    5797 scope.go:122] "RemoveContainer" containerID="fcf2cf02fb799c670b0ed5ce89930ab6d170d3977f6ac0623702a7651585c4d1"
Nov 04 01:22:36.735305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:36.707125    5797 scope.go:122] "RemoveContainer" containerID="94815c2c789983c778f9abd229c5af4dbd56142d74e74c06d396009bc9c9a75f"
Nov 04 01:22:36.777614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:36.777587    5797 scope.go:122] "RemoveContainer" containerID="9ad2d4360864c677cb84d54342534ddfd9266a8a47ece63b3bb085f7e336d5c9"
Nov 04 01:22:36.810314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:36.809967    5797 scope.go:122] "RemoveContainer" containerID="69e837bc1683ed2ea071a07beb61ea66c71261dbec47f848508076f19b5629d6"
Nov 04 01:22:36.824895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:36.824874    5797 scope.go:122] "RemoveContainer" containerID="2bff4fc5c2842c836d8194ecdc26a6cc4c3bf4fd6a216efe5d85f3088fe8ceb4"
Nov 04 01:22:37.065380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:37.029776    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"]
Nov 04 01:22:37.065380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:37.037044    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4b84d3-2fntb"]
Nov 04 01:22:37.308081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:37.305347    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:37.701654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:37.701609    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"]
Nov 04 01:22:37.824782 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:22:37.824743    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec4f014_0a81_40fb_82f9_6b1c291b4f0c.slice/crio-acea860056062f3d22011433f8c11081552107852cc26011f355b3111c132f50 WatchSource:0}: Error finding container acea860056062f3d22011433f8c11081552107852cc26011f355b3111c132f50: Status 404 returned error can't find the container with id acea860056062f3d22011433f8c11081552107852cc26011f355b3111c132f50
Nov 04 01:22:38.112935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:38.112739    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:38.425015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:38.421537    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:22:38.647116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:38.526319    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3107ff-b7e8-4895-a996-3846ea3b7f8d" path="/var/lib/kubelet/pods/fc3107ff-b7e8-4895-a996-3846ea3b7f8d/volumes"
Nov 04 01:22:38.798014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:38.797274    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p" event={"ID":"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c","Type":"ContainerStarted","Data":"acea860056062f3d22011433f8c11081552107852cc26011f355b3111c132f50"}
Nov 04 01:22:39.166458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:39.166386    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:39.809650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:39.809582    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p" event={"ID":"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c","Type":"ContainerStarted","Data":"31df58fc388c16e5b26f6cc3a15af78356624c175d8528070ff141a7a97c5994"}
Nov 04 01:22:39.877817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:39.876597    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p" podStartSLOduration=10.876574927 podStartE2EDuration="10.876574927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:39.866555846 +0000 UTC m=+1147.508462123" watchObservedRunningTime="2026-11-04 01:22:39.876574927 +0000 UTC m=+1147.518481179"
Nov 04 01:22:40.037020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:40.036922    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:40.251013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:40.247170    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:40.251013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:40.247256    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:41.024626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.024588    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"]
Nov 04 01:22:41.031330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.031300    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57366fe0-wd22h"]
Nov 04 01:22:41.234368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.230589    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:41.829913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.829878    5797 generic.go:356] "Generic (PLEG): container finished" podID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3" exitCode=0
Nov 04 01:22:41.830355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.829955    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" event={"ID":"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38","Type":"ContainerDied","Data":"38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"}
Nov 04 01:22:41.830658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.830638    5797 scope.go:122] "RemoveContainer" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:22:41.832019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.831998    5797 generic.go:356] "Generic (PLEG): container finished" podID="8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" containerID="31df58fc388c16e5b26f6cc3a15af78356624c175d8528070ff141a7a97c5994" exitCode=0
Nov 04 01:22:41.832110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.832047    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p" event={"ID":"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c","Type":"ContainerDied","Data":"31df58fc388c16e5b26f6cc3a15af78356624c175d8528070ff141a7a97c5994"}
Nov 04 01:22:41.832495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.832467    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:22:41.833836 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.833814    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:22:41.833926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.833878    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:22:41.883245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:41.883212    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:42.182592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.182510    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:42.312132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.311946    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"]
Nov 04 01:22:42.330170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.330122    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.335631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.335218    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6e609c-openshift-default-dockercfg-n2474\""
Nov 04 01:22:42.339650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.339446    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"]
Nov 04 01:22:42.439041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.438945    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:42.441959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.441934    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442029    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442055    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442199    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442251    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442354    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcspj\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442444    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442507    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.442935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.442536    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.543841    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.543973    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544027    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544082    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544146    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544206    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcspj\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544313    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544368    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.544397    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.545019    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.545279    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.549009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.545882    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.551739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.551690    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.552254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.552232    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.623011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.596599    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcspj\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.623011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.609508    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.660205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.660168    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:42.672039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.663634    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.696692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.674359    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc128a7-238a-4041-b3f8-a5ad95d97ee1" path="/var/lib/kubelet/pods/0fc128a7-238a-4041-b3f8-a5ad95d97ee1/volumes"
Nov 04 01:22:42.697469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.697431    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:42.919491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.919449    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" event={"ID":"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38","Type":"ContainerStarted","Data":"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"}
Nov 04 01:22:42.994031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.980305    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:22:42.994031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.980425    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:42.994031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.980480    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:42.994672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:42.994648    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:43.239232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.238678    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:43.303011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.302615    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca\") pod \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") "
Nov 04 01:22:43.303011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.302728    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca\") pod \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") "
Nov 04 01:22:43.303011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.302918    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq\" (UniqueName: \"kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq\") pod \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\" (UID: \"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c\") "
Nov 04 01:22:43.315472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.314575    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca" pod "8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" (UID: "8ec4f014-0a81-40fb-82f9-6b1c291b4f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:22:43.320942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.320331    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca" pod "8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" (UID: "8ec4f014-0a81-40fb-82f9-6b1c291b4f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:22:43.321126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.321064    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq" pod "8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" (UID: "8ec4f014-0a81-40fb-82f9-6b1c291b4f0c"). InnerVolumeSpecName "kube-api-access-z9wbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:22:43.612901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.594290    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:43.612901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.606889    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9wbq\" (UniqueName: \"kubernetes.io/projected/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-kube-api-access-z9wbq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:22:43.612901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.606918    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:22:43.612901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.606934    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:22:43.638624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.627892    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"]
Nov 04 01:22:43.646852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.646220    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:22:43.646852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.646313    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:22:43.646852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.646352    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:22:43.931049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.930912    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p" event={"ID":"8ec4f014-0a81-40fb-82f9-6b1c291b4f0c","Type":"ContainerDied","Data":"acea860056062f3d22011433f8c11081552107852cc26011f355b3111c132f50"}
Nov 04 01:22:43.931049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.930985    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acea860056062f3d22011433f8c11081552107852cc26011f355b3111c132f50"
Nov 04 01:22:43.931966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.931947    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"
Nov 04 01:22:43.937025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.935278    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" event={"ID":"af20f3c0-5693-4ff4-9587-616247f7553e","Type":"ContainerStarted","Data":"cc2b338eb804b3fd1e008b956c0d923b7f15cace16a55089688388df0ddfabea"}
Nov 04 01:22:43.937025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.935330    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" event={"ID":"af20f3c0-5693-4ff4-9587-616247f7553e","Type":"ContainerStarted","Data":"5e1c67de66a5f397808634d578e46fa3313486407d196e9b8eacba815b1f2fa6"}
Nov 04 01:22:43.991040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.983069    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:43.991040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.983125    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:43.991040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.984026    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podStartSLOduration=1.984007723 podStartE2EDuration="1.984007723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:43.959537188 +0000 UTC m=+1151.601443464" watchObservedRunningTime="2026-11-04 01:22:43.984007723 +0000 UTC m=+1151.625913967"
Nov 04 01:22:44.009076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:43.999089    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:44.617646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:44.610316    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:44.960076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:44.959079    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:44.960076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:44.959155    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:45.005668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:45.005635    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:45.525013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:45.520800    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:45.959099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:45.958601    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:45.959099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:45.958696    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:45.976622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:45.976598    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:45.978780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:45.978747    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:22:46.561015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:46.556733    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:46.766017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:46.764543    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:46.960508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:46.960462    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:46.960782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:46.960528    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:47.031260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:47.031220    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"]
Nov 04 01:22:47.035704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:47.035606    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89b2f07-bplc2"]
Nov 04 01:22:47.789286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:47.789220    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:47.964448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:47.964405    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:47.964729 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:47.964696    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:48.102587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:48.102491    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:48.491408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:48.491375    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178c4ef8-17dd-41bf-b73e-66a53ef646c0" path="/var/lib/kubelet/pods/178c4ef8-17dd-41bf-b73e-66a53ef646c0/volumes"
Nov 04 01:22:48.982017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:48.960458    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 04 01:22:48.982017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:48.960537    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 04 01:22:49.037121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.020933    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:49.116065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.113282    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"]
Nov 04 01:22:49.124066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.123970    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"]
Nov 04 01:22:49.130931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.130900    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3af645a-h4svc"]
Nov 04 01:22:49.137027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.136982    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e4eefc-hhvw5"]
Nov 04 01:22:49.623706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.623662    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:50.006064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:49.998840    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:50.017231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:50.017209    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:22:50.018201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:50.018175    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:22:50.019004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:50.018971    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:22:50.506852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:50.506807    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77353796-050c-449b-9614-a778f6c68f3e" path="/var/lib/kubelet/pods/77353796-050c-449b-9614-a778f6c68f3e/volumes"
Nov 04 01:22:50.507652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:50.507626    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fa7fcc-738c-4e55-b73a-859b679f501a" path="/var/lib/kubelet/pods/79fa7fcc-738c-4e55-b73a-859b679f501a/volumes"
Nov 04 01:22:50.902928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:50.902886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:51.031503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:51.031466    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"]
Nov 04 01:22:51.037822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:51.037790    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75ddfb3-zcj99"]
Nov 04 01:22:51.975219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:51.975181    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:52.185357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:52.185320    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:52.555322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:52.555283    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f17b85f-4d63-4903-9a1c-9bd4c516be8c" path="/var/lib/kubelet/pods/2f17b85f-4d63-4903-9a1c-9bd4c516be8c/volumes"
Nov 04 01:22:53.025716 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:53.025674    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"]
Nov 04 01:22:53.029628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:53.029597    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca75a03-tl64z"]
Nov 04 01:22:53.526749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:53.526543    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:54.388423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:54.388389    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:54.550820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:54.550782    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:22:54.654590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:54.654516    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:54.943760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:54.943686    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df782c7-7aab-4768-935d-944a5bb83068" path="/var/lib/kubelet/pods/1df782c7-7aab-4768-935d-944a5bb83068/volumes"
Nov 04 01:22:55.248229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:55.248140    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:22:55.248496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:55.248222    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:22:55.542806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:55.542681    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:55.764265 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:55.764229    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:56.303290 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.303168    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:56.408005 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.407959    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"]
Nov 04 01:22:56.662023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.661979    5797 cpu_manager.go:414] "containerMap: removing container" podUID="8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" containerName="revoke-keys"
Nov 04 01:22:56.662023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.662023    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" containerName="revoke-keys"
Nov 04 01:22:56.662588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.662325    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" containerName="revoke-keys"
Nov 04 01:22:56.767887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.767845    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"]
Nov 04 01:22:56.768087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.768026    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:56.773159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.773135    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:22:56.773315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.773229    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:22:56.827033 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:22:56.826554    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:22:56.918647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.918565    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhqw\" (UniqueName: \"kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:56.918884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.918811    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:56.920111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:56.920071    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.052011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.037737    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhqw\" (UniqueName: \"kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.052011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.038016    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.052011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.038328    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.052011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.039483    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.111029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.110977    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:57.190437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.190350    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.208752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.190522    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhqw\" (UniqueName: \"kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw\") pod \"maas-api-revoke-sub-cf4ac610-7tzpr\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.440528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.440458    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:22:57.794078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:57.793921    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:58.496066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:58.495272    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:22:58.979205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:58.979165    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"]
Nov 04 01:22:59.019958 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:22:59.007432    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a659a7b_4ca9_499e_8059_d6916330925a.slice/crio-a45c3f2bd7c2d6f27aab56d76bd3b268acbf3aeec0548010b47c0a49fb8fa8d6 WatchSource:0}: Error finding container a45c3f2bd7c2d6f27aab56d76bd3b268acbf3aeec0548010b47c0a49fb8fa8d6: Status 404 returned error can't find the container with id a45c3f2bd7c2d6f27aab56d76bd3b268acbf3aeec0548010b47c0a49fb8fa8d6
Nov 04 01:22:59.230522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:59.230435    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr" event={"ID":"8a659a7b-4ca9-499e-8059-d6916330925a","Type":"ContainerStarted","Data":"a45c3f2bd7c2d6f27aab56d76bd3b268acbf3aeec0548010b47c0a49fb8fa8d6"}
Nov 04 01:22:59.438118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:22:59.436677    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:00.246395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:00.246355    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr" event={"ID":"8a659a7b-4ca9-499e-8059-d6916330925a","Type":"ContainerStarted","Data":"5468be55aef0f918df758a401ab84477feaae2529c3dfefdfdb04d15783d67e1"}
Nov 04 01:23:00.271088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:00.271033    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr" podStartSLOduration=4.271014787 podStartE2EDuration="4.271014787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:00.268249424 +0000 UTC m=+1167.910155699" watchObservedRunningTime="2026-11-04 01:23:00.271014787 +0000 UTC m=+1167.912921036"
Nov 04 01:23:00.479218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:00.478301    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:01.079812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:01.079715    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"]
Nov 04 01:23:01.090412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:01.090374    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6b8db9-vjlrg"]
Nov 04 01:23:01.257557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:01.257524    5797 generic.go:356] "Generic (PLEG): container finished" podID="8a659a7b-4ca9-499e-8059-d6916330925a" containerID="5468be55aef0f918df758a401ab84477feaae2529c3dfefdfdb04d15783d67e1" exitCode=0
Nov 04 01:23:01.258128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:01.257607    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr" event={"ID":"8a659a7b-4ca9-499e-8059-d6916330925a","Type":"ContainerDied","Data":"5468be55aef0f918df758a401ab84477feaae2529c3dfefdfdb04d15783d67e1"}
Nov 04 01:23:01.414014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:01.407200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:02.268681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.268643    5797 generic.go:356] "Generic (PLEG): container finished" podID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerID="f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e" exitCode=0
Nov 04 01:23:02.269871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.268708    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" event={"ID":"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe","Type":"ContainerDied","Data":"f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"}
Nov 04 01:23:02.271473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.270573    5797 scope.go:122] "RemoveContainer" containerID="f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"
Nov 04 01:23:02.272455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.272424    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:02.272549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.272496    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:02.272549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.272527    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:02.455438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.399548    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:02.561685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.561617    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70346d7c-b077-481d-a38a-64489f7caea3" path="/var/lib/kubelet/pods/70346d7c-b077-481d-a38a-64489f7caea3/volumes"
Nov 04 01:23:02.634837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.634802    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:02.787984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:02.786790    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:23:03.056392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.056298    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca\") pod \"8a659a7b-4ca9-499e-8059-d6916330925a\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") "
Nov 04 01:23:03.057014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.056959    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca\") pod \"8a659a7b-4ca9-499e-8059-d6916330925a\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") "
Nov 04 01:23:03.058107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.057360    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca" pod "8a659a7b-4ca9-499e-8059-d6916330925a" (UID: "8a659a7b-4ca9-499e-8059-d6916330925a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:23:03.058107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.057584    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw\" (UniqueName: \"kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw\") pod \"8a659a7b-4ca9-499e-8059-d6916330925a\" (UID: \"8a659a7b-4ca9-499e-8059-d6916330925a\") "
Nov 04 01:23:03.059598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.059568    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:23:03.185185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.185142    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw" pod "8a659a7b-4ca9-499e-8059-d6916330925a" (UID: "8a659a7b-4ca9-499e-8059-d6916330925a"). InnerVolumeSpecName "kube-api-access-jdhqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:23:03.212169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.212130    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca" pod "8a659a7b-4ca9-499e-8059-d6916330925a" (UID: "8a659a7b-4ca9-499e-8059-d6916330925a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:23:03.269898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.268939    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a659a7b-4ca9-499e-8059-d6916330925a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:23:03.269898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.268982    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhqw\" (UniqueName: \"kubernetes.io/projected/8a659a7b-4ca9-499e-8059-d6916330925a-kube-api-access-jdhqw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:23:03.285583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.285546    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" event={"ID":"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe","Type":"ContainerStarted","Data":"2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051"}
Nov 04 01:23:03.287428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.287405    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"
Nov 04 01:23:03.287595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.287433    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr" event={"ID":"8a659a7b-4ca9-499e-8059-d6916330925a","Type":"ContainerDied","Data":"a45c3f2bd7c2d6f27aab56d76bd3b268acbf3aeec0548010b47c0a49fb8fa8d6"}
Nov 04 01:23:03.287595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.287489    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45c3f2bd7c2d6f27aab56d76bd3b268acbf3aeec0548010b47c0a49fb8fa8d6"
Nov 04 01:23:03.310472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.310392    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:23:03.310647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.310596    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:03.311528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.310660    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:03.594066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:03.593636    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:04.173840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:04.173797    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:04.310715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:04.310669    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:04.311222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:04.310728    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:04.555630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:04.555496    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:05.315712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:05.315672    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:05.316268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:05.315757    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:05.510019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:05.506014    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:06.310013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:06.309588    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:06.319012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:06.315448    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:06.319012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:06.315526    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:06.776884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:06.776847    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:07.189012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:07.188830    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:07.309943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:07.309910    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:07.310187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:07.309977    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:07.986635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:07.986600    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:08.310169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:08.310080    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:08.310169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:08.310137    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:08.876013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:08.875529    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:09.207283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:09.207246    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:23:09.309958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:09.309924    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 04 01:23:09.310168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:09.310027    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 04 01:23:09.605945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:09.605849    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:10.313066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:10.313025    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:10.330012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:10.326568    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:23:10.330012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:10.329802    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:23:11.186610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:11.186571    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:11.875026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:11.874972    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:12.673097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:12.673064    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:12.900004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:12.899964    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:13.298089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:13.298054    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:13.744256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:13.744222    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:13.991568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:13.991527    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:14.775972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:14.774872    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:15.570251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:15.570214    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:16.034628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:16.034592    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:16.254584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:16.254548    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:16.457961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:16.457929    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:17.547018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:17.543339    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:18.288241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:18.288162    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:18.997928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:18.997827    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:19.517770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:19.517728    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:19.798089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:19.796178    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:20.528433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:20.528392    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:21.307725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:21.307678    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:22.050026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:22.045059    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"]
Nov 04 01:23:22.079014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:22.076483    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ea69b7-qw598"]
Nov 04 01:23:22.100450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:22.100412    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:22.345217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:22.345133    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:22.480175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:22.480140    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9521b723-3f93-4e36-8c9c-c415cf6f82e1" path="/var/lib/kubelet/pods/9521b723-3f93-4e36-8c9c-c415cf6f82e1/volumes"
Nov 04 01:23:22.989211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:22.989175    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:23.031442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.031394    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"]
Nov 04 01:23:23.049057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.048952    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be8faa-cz7gf"]
Nov 04 01:23:23.249449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.249361    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:23.352845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.352809    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"]
Nov 04 01:23:23.354873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.354846    5797 cpu_manager.go:414] "containerMap: removing container" podUID="8a659a7b-4ca9-499e-8059-d6916330925a" containerName="revoke-keys"
Nov 04 01:23:23.355049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.355034    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a659a7b-4ca9-499e-8059-d6916330925a" containerName="revoke-keys"
Nov 04 01:23:23.356218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.356197    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a659a7b-4ca9-499e-8059-d6916330925a" containerName="revoke-keys"
Nov 04 01:23:23.375262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.375226    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.383708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.383678    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"]
Nov 04 01:23:23.390490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.390465    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-20a250-openshift-default-dockercfg-qqxws\""
Nov 04 01:23:23.488278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.488246    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545273    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545344    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545479    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545626    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545698    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzb2q\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545751    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545847    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.545919    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.552076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.546112    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.669507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.669465    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.670971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.670942    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzb2q\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.684199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.684157    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.685114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.685090    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.685814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.685792    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.686182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.686152    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.687049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.686346    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.687049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.686395    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.687049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.686555    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.687049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.686810    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.691174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.688103    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.691174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.688317    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.691174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.684816    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.691174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.688979    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.707251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.706375    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:23.718538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.718505    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.732773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.732703    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.733091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.733071    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:23.733769 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:23.733738    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzb2q\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q\") pod \"e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") " pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:24.003162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.002658    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:24.049763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.032019    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"]
Nov 04 01:23:24.049981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.049777    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141e1f6-jhtdd"]
Nov 04 01:23:24.078901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.078515    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:24.259610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.259515    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"]
Nov 04 01:23:24.267257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.267229    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.273648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.273624    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fdd516-openshift-default-dockercfg-452l6\""
Nov 04 01:23:24.279561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.279496    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"]
Nov 04 01:23:24.341567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.341482    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.341733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.341613    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.341733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.341673    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.341733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.341700    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.343365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.341832    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.343365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.341932    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.343365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.342027    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.343365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.342086    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.343365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.342211    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vf5\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458269    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vf5\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458397    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458452    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458509    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458534    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458673    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458777    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458846    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.458905    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.469103    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.469632    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.474017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.470327    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.511612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.509108    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:24.511612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.509538    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.511612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.510287    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.652674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.652473    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d5a96d-dd44-4c20-a261-d338349c6c83" path="/var/lib/kubelet/pods/29d5a96d-dd44-4c20-a261-d338349c6c83/volumes"
Nov 04 01:23:24.688010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.675096    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85280e39-4b0c-4515-8889-e81e6e77ee3b" path="/var/lib/kubelet/pods/85280e39-4b0c-4515-8889-e81e6e77ee3b/volumes"
Nov 04 01:23:24.689208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.689172    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.692168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.690481    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.692168 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:23:24.691290    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9ad66a_bede_4e9a_94ea_f9dd9a924f61.slice/crio-861fc64134ffaa424734c20a32822a3ed98687e1a2ad9852c4b3a8941f944214 WatchSource:0}: Error finding container 861fc64134ffaa424734c20a32822a3ed98687e1a2ad9852c4b3a8941f944214: Status 404 returned error can't find the container with id 861fc64134ffaa424734c20a32822a3ed98687e1a2ad9852c4b3a8941f944214
Nov 04 01:23:24.692168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.691329    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"]
Nov 04 01:23:24.693330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.693253    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vf5\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.698790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.698747    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:24.698902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.698847    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:24.698902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.698893    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:24.699446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.699423    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token\") pod \"e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") " pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:24.712802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.712554    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" event={"ID":"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61","Type":"ContainerStarted","Data":"861fc64134ffaa424734c20a32822a3ed98687e1a2ad9852c4b3a8941f944214"}
Nov 04 01:23:24.905824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.903860    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:24.905824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:24.904226    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:25.270024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.262690    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:25.270024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.269158    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"]
Nov 04 01:23:25.275749 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:23:25.272704    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f3f13f_1942_4dca_b993_0f48deaa503e.slice/crio-af22ad3c60d50c2f41cceff6dd15a813fc63384e4855b14ec8bd42798d3b42eb WatchSource:0}: Error finding container af22ad3c60d50c2f41cceff6dd15a813fc63384e4855b14ec8bd42798d3b42eb: Status 404 returned error can't find the container with id af22ad3c60d50c2f41cceff6dd15a813fc63384e4855b14ec8bd42798d3b42eb
Nov 04 01:23:25.281789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.281735    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:25.282068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.282025    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:25.282186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.282087    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:23:25.724293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.724256    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" event={"ID":"33f3f13f-1942-4dca-b993-0f48deaa503e","Type":"ContainerStarted","Data":"9c02dd011ff8f37cdc4a2431185502d486e3d8d2b0cc5c4c685bd627b3bf06f3"}
Nov 04 01:23:25.724509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.724323    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" event={"ID":"33f3f13f-1942-4dca-b993-0f48deaa503e","Type":"ContainerStarted","Data":"af22ad3c60d50c2f41cceff6dd15a813fc63384e4855b14ec8bd42798d3b42eb"}
Nov 04 01:23:25.726394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.726363    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" event={"ID":"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61","Type":"ContainerStarted","Data":"275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"}
Nov 04 01:23:25.751893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.751834    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podStartSLOduration=1.7518172939999999 podStartE2EDuration="1.751817294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:25.746963625 +0000 UTC m=+1193.388869877" watchObservedRunningTime="2026-11-04 01:23:25.751817294 +0000 UTC m=+1193.393723546"
Nov 04 01:23:25.772111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.772055    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podStartSLOduration=2.772041536 podStartE2EDuration="2.772041536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:25.768855536 +0000 UTC m=+1193.410761787" watchObservedRunningTime="2026-11-04 01:23:25.772041536 +0000 UTC m=+1193.413947785"
Nov 04 01:23:25.908287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.908250    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:25.990128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:25.987547    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:26.016035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:26.009933    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:26.026236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:26.026204    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:26.254737 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:23:26.254652    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:23:26.693741 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:26.693233    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:26.736802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:26.736767    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:26.737587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:26.737566    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:23:26.913902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:26.913868    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:27.675014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:27.674514    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:27.747289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:27.747254    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:27.748202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:27.748179    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:23:28.375207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:28.375171    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:28.617237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:28.617203    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:28.833757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:28.833666    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:29.483471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:29.483430    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:29.762796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:29.746887    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:30.383267 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:30.383234    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:30.599289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:30.599251    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:30.946348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:30.946307    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:31.214220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:31.213344    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:31.802982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:31.802943    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:32.006413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:32.006373    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:32.330712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:32.330677    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:32.592422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:32.592330    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:32.952387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:32.952351    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:33.437018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:33.430344    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:34.275373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:34.275339    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:35.078484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:35.078431    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:35.818956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:35.818920    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:36.626025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:36.625889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:37.146397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.146364    5797 scope.go:122] "RemoveContainer" containerID="4a54c15163995711a617173e3afdae3a948ad2a710c7175f0b1152a9491651d8"
Nov 04 01:23:37.165441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.165412    5797 scope.go:122] "RemoveContainer" containerID="c76c2bae4055926baf8c01297fbbbe5755444f65db979642e69ed299795c2c7c"
Nov 04 01:23:37.207063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.207035    5797 scope.go:122] "RemoveContainer" containerID="f3ea11000c609ca51945a9d51a1e898b96c2fe53bd2363a0588025db42b7ac1f"
Nov 04 01:23:37.220316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.220291    5797 scope.go:122] "RemoveContainer" containerID="479b0dc99c87b2c71dfeb8125b5ae9195bc77b651a6997bbb8d78fcfb3ce39f7"
Nov 04 01:23:37.231180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.231160    5797 scope.go:122] "RemoveContainer" containerID="033c923ca7e1471ea227770dc2815e7ca0908b4c625eb43c6b9959826bcdeeec"
Nov 04 01:23:37.248865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.243930    5797 scope.go:122] "RemoveContainer" containerID="09bece2c3294c6c4755a7e13918f863d606b7ddef18985285e263956f65bba71"
Nov 04 01:23:37.257867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.257844    5797 scope.go:122] "RemoveContainer" containerID="da31062d1c506258f98f5bd48c0b2d689cc5ed9753774374a5888420e41f17d9"
Nov 04 01:23:37.271499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.271471    5797 scope.go:122] "RemoveContainer" containerID="5a7bce3d2f9f24213d1c23b328b8759542abcf2cab9be0ed401d626481531f17"
Nov 04 01:23:37.283329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.283307    5797 scope.go:122] "RemoveContainer" containerID="21867df1877375196ba601c1f58d177e0bfc9ac5bc2ce33f016d972fbd51f6c4"
Nov 04 01:23:37.295053 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.295030    5797 scope.go:122] "RemoveContainer" containerID="21935c3a12d86eddea14a460bbc320be97830d0c42ab1d83d5d8e9c6b1e46b75"
Nov 04 01:23:37.306710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.306683    5797 scope.go:122] "RemoveContainer" containerID="6919b84e26580a64da23a664e962203d49fc67a8f4cc2a17e0136b780fc411de"
Nov 04 01:23:37.319406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.319379    5797 scope.go:122] "RemoveContainer" containerID="9244877619ef53e0766dcb5da0759296fd4f96828fadc44fbd741f1c16e049f5"
Nov 04 01:23:37.333475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.333446    5797 scope.go:122] "RemoveContainer" containerID="d2aac2ad7d9a42082588c3f1f3034f90ed159eca313c8e987981a0fb4ad7648f"
Nov 04 01:23:37.344109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.344089    5797 scope.go:122] "RemoveContainer" containerID="0378315dbee705c60b46ad1e3b849e8b79287b4c5841adbdd577298da1a6dc3c"
Nov 04 01:23:37.355331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.355312    5797 scope.go:122] "RemoveContainer" containerID="d01bdcaee6f20cfa8db924de1c7e5722c845c6f9a5f6829a83ce5d9fe03a352b"
Nov 04 01:23:37.492075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:37.489398    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:38.285019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:38.279456    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:38.985170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:38.985120    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:39.285290 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:39.285174    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:39.775051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:39.767817    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:40.037857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:40.037775    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"]
Nov 04 01:23:40.042465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:40.042435    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba0ba93-cmq5n"]
Nov 04 01:23:40.508405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:40.508367    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a88e516-f86a-451f-9645-619b26d1eb86" path="/var/lib/kubelet/pods/6a88e516-f86a-451f-9645-619b26d1eb86/volumes"
Nov 04 01:23:40.901949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:40.901914    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:42.613138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.610731    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:42.613138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.611736    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:23:42.635165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.635121    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:42.674101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.673956    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:42.763014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.758970    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:23:42.871478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.870847    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:42.872983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:42.872941    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:43.118015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.117290    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.556703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.552853    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.556703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.554674    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.615305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.615274    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:23:43.616050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.615207    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.617036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.616363    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:23:43.617812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.617793    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:23:43.645307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.637234    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.645956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.645936    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:23:43.684168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.680061    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.684168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.680638    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:23:43.770224 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.770143    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:23:43.802854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.802822    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:23:43.889110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.841115    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.889110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.875711    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:23:43.889110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.875772    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:43.889110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.876064    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:23:43.947159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:43.900289    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:23:44.529464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.513555    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:44.567022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.557574    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:44.577059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.571273    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:44.636176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.636118    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:44.778062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.663346    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:44.778062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.663397    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:44.808118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:44.773229    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:23:45.015012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.010884    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:23:45.075077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.068407    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:45.235015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.201180    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:45.530011 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:23:45.526743    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.014s"
Nov 04 01:23:45.545081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.538029    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:23:45.545081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.538498    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:23:45.575522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.575471    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:45.622282 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.622236    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:45.794069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.793346    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:23:45.794554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.794406    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:45.844180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.826066    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:23:45.935013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:45.925127    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:23:46.026488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.026439    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:46.202239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.202192    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:46.219290 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.219260    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:23:46.238510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.238482    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:23:46.296340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.296310    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:23:46.296586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.296575    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:23:46.300606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.300570    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:23:46.319734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.319704    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:23:46.333436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.333404    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:23:46.333748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.333728    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:23:46.505263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.505160    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:46.982026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:46.967877    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:47.197569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:47.197527    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:47.397129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:47.397087    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:48.284119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:48.284084    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:48.500619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:48.500575    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:50.055490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:50.055400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:51.093937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:51.093892    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:51.744013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:51.739341    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:52.687012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:52.676539    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:53.401042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:53.397551    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:54.179774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:54.179686    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:54.968821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:54.967240    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:55.659045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:55.659013    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:55.876322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:55.876275    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:56.156820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:56.156669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:56.525343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:56.516285    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:57.184731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:57.184690    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:57.384056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:57.384025    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:23:59.468592 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:23:59.468315    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:23:59.627242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:59.402256    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:23:59.627242 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:23:59.494485    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:00.083235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.083188    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"]
Nov 04 01:24:00.327754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.327721    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.383537    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.383656    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndhr\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.383715    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.383875    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.384048    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.384082    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.384118    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.384203    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.441505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.384280    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486423    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486507    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndhr\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486542    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486626    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486771    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486823    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486872    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486905    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.486958    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.487108    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.487426    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.487482    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.501396    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.610979    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.610979    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:00.946253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:00.944203    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.044125    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.078714    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.079209    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.613284    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.613481    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.781531    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:01.781589    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndhr\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr\") pod \"e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") " pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.018145    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.018217    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.025552    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.120124    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.132047    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:24:02.397813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.251480    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.681876    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.681947    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.900479    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.914115    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:02.914171    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:03.511113    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:06.264587    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:06.277522    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:06.277579    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.019825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:06.283826    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.483903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:07.483858    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded"
Nov 04 01:24:07.680625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:07.680578    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:07.685216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:07.685192    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 04 01:24:07.685725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:07.685705    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:24:07.686303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:07.686283    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:07.686359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:07.686322    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:11.550116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:09.242403    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:24:11.550116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:09.243085    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.046791    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.193081    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.193404    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.193715    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.194203    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.195719    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.196288    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.197609    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.198280    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.223213    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.231222    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.418150    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.419425    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:34.885287    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Liveness probe status=failure output="Get \"https://10.134.0.18:8443/health\": context deadline exceeded" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.420234    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.195690    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.421304    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.421584    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.199782    5797 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-597546666c-728zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.421897    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podUID="0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.404611    5797 trace.go:236] Trace[183683784]: "Calculate volume metrics of workload-certs for pod openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" (04-Nov-2026 01:24:31.934) (total time: 3469ms):
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: Trace[183683784]: [3.469881856s] [3.469881856s] END
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:33.973169    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.422287    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.429345    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjqs2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.429378    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.429593    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.429639    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.457687    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.475483    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.632636    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:35.632695    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.112599    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.112743    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.112774    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.112804    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.112871    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.112892    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.134447    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:50052\" within 1s: context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.134606    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.134639    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.134740    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.134754    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.193294    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.194371    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.194390    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.196094    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.197624    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.197719    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.198730    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.201965    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.202042    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.209218    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.211363    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.211366    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.214777    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.216745    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.224258    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.229399    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.229453    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.263123    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.263186    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.422135    5797 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-597546666c-728zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.422144    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.422256    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.422347    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podUID="0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.457561    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.457633    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.460710    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.463118    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.463155    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.471109    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.471166    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.475886    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.516115    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:24:35.963645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:36.516177    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.113803    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.113803    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.114894    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.114950    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.194289    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.195359    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.195367    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.196450    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.198607    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.198631    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.199836    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.203099    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.203146    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.210330    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.212464    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.212493    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.215623    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.217755    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.224912    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.230142    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.230206    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.423336    5797 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-597546666c-728zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.423410    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podUID="0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.458537    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.458606    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.461687    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:37.476874    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.114627    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.194865    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.195949    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.195947    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.197059    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.199273    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.200044    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.210584    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.225931    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:38.852860    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:41.550252    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:41.550414    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:41.550432    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:41.550501    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:43.302054    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:43.429269    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:44.436823    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r7v8r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:44.436911    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.90:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:47.121093    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.286216    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.286318    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.286415    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.608615    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.648803    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.648864    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:51.964709    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:52.177409    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:52.487127    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:52.488418    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:52.989095    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:52.989156    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.004202    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.004259    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.042749    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.044115    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.51:50052: i/o timeout"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.046771    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeNotReady"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.046799    5797 setters.go:546] "Node became not ready" node="ip-10-0-140-186.us-east-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T01:24:53Z","lastTransitionTime":"2026-11-04T01:24:53Z","reason":"KubeletNotReady","message":"container runtime is down"}
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.130803    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.149494    5797 patch_prober.go:29] interesting pod/dns-default-z88mh container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.149551    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-z88mh" podUID="b1de55b1-797e-4e61-aeee-624aa4e20ca5" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.151169    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.151221    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.156426    5797 trace.go:236] Trace[1372794289]: "Calculate volume metrics of insights-runtime-extractor-tls for pod openshift-insights/insights-runtime-extractor-lbqr9" (04-Nov-2026 01:24:48.340) (total time: 4617ms):
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: Trace[1372794289]: [4.617117249s] [4.617117249s] END
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.210758    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.215667    5797 trace.go:236] Trace[1601411059]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" (04-Nov-2026 01:24:51.650) (total time: 1564ms):
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: Trace[1601411059]: [1.564879214s] [1.564879214s] END
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.215934    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.215967    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.222130    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.992039057s: [/var/lib/containers/storage/overlay/df68cb69434860319cc5cce7813527dfae5f27e5d5b48c1f8fde2347d90553c4/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.222340    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.99227215s: [/var/lib/containers/storage/overlay/b02a66a740d34612a3e71f9c1aede7e78780fab3148c9286cab0a682d80b71be/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.222366    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 45.753917546s: [/var/lib/containers/storage/overlay/d4f2348b3bc78798b579e1d33bbdea20605d5dffc3c92838c8a40806899ed387/diff /var/log/pods/e2e-models-e2e-worker-w1-1036e8_simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25_9a49d108-2491-4c6c-808d-3b4fd1015e88/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.223144    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.223182    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.225181    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.993379917s: [/var/lib/containers/storage/overlay/16718ce101b9c26c0830810c5a35aca438d77162eb9398d4b20a731a7269ea68/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.228539    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.229135    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.230528    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.230561    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.241456    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.241493    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.241950    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.242023    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.242272    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.242319    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.242826    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.242869    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.243966    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.855767174s: [/var/lib/containers/storage/overlay/a5feaaedfdc18ccd75d803111ca81704fc483ee8e60c467d9e28055520a93e8a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.244025    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.839926804s: [/var/lib/containers/storage/overlay/f79ba5de714466388e5d1ee630b8ceded3a7c30cf2e7e6b00993f50f055c5733/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.248331    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.248364    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.262892    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.262925    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.283722    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879317994s: [/var/lib/containers/storage/overlay/a2f3bbdba38158abf8924e72873c44734a70baee9fc4b1f25186e1e4ff6343d5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284150    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879745082s: [/var/lib/containers/storage/overlay/8ce89f610c6de04b8ff9e104b7ba9b1bbab0887598d8a9fc3bc80853980ac667/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284177    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879773477s: [/var/lib/containers/storage/overlay/833dae95a068a6a2cac250e8f3b580aeb0ff5e63070cf644cb148fd1f0da19b9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284201    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879788674s: [/var/lib/containers/storage/overlay/8f88dda44c1fb05ce59dc2e9a46d853fec73a2e3045dde039f07bcc592ef7ee0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284237    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879823415s: [/var/lib/containers/storage/overlay/6a3249d9e478124a689e2122fb0bffb1f0b3ea9aa64328cd3d5871f85f76502f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284264    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879849847s: [/var/lib/containers/storage/overlay/9ca311bdff2fb2f60791f6f0452569b3d91e31c44f6da8ccc69d44d6ba95dbcf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284293    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879876954s: [/var/lib/containers/storage/overlay/5a2177d0c02a7a210cff80e99855e203223cdfd760c49b1c98fe1c4d1b872c8f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284327    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.8799091s: [/var/lib/containers/storage/overlay/4cdca34da796ee408074d9c452dc6d1c85af035582518d0c003892527f7c4640/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284359    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879940541s: [/var/lib/containers/storage/overlay/fb0decfb35a78a1f615d60b4b79c8a62e80a59c65499ca14c024ed16fcb908ba/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284392    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.879973118s: [/var/lib/containers/storage/overlay/03c52ff807bb164a8436941e71b2372efe7984a5c0f1774dd8c70840aa124214/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284504    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880080155s: [/var/lib/containers/storage/overlay/e62b78b9615627356de9ff5e3344709045bcec7fdf2b03c31a9b08401ecfdeb9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284536    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.88011331s: [/var/lib/containers/storage/overlay/971efd28ccd77c16533657812ab39ae180cef64f99bce196abbc71a0bebadd3f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284567    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880142519s: [/var/lib/containers/storage/overlay/9fb15a7070fb70dfb8296806f4d3188fe574b509ddc8af7c0275551eeceed785/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284596    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880171072s: [/var/lib/containers/storage/overlay/c7fe3dc096b833b3ff79964e7fc1bc2e0661703b8e4de8ccd662e2306a82546e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284620    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880189391s: [/var/lib/containers/storage/overlay/f8a65bed28bd5ad3742f26c2f7abaf25511529eeb9b2be9e392210f6926071c7/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284643    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880211546s: [/var/lib/containers/storage/overlay/e566206769ba765fd7edf3a467a8db569786a2b09055eba3b277cb0ac5ce4740/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284666    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880233287s: [/var/lib/containers/storage/overlay/43694b66c5c8d7de1002525da039574bf6f0894e5371986a77602dde5a3c25ab/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284690    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880255916s: [/var/lib/containers/storage/overlay/83faa1b7c3e2668ce3c5462f79237bf89672068207c648d681afcf7401721320/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284713    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880277387s: [/var/lib/containers/storage/overlay/79e196c97318c8928a1b03fda6b41f4b122c25462f6f55a6c608900067aafb87/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284737    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880300678s: [/var/lib/containers/storage/overlay/f7a7eaf0d469e60fd74750252a783f7d46b509f50ccd3e45ec05f62d5e73bd0a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284840    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880401857s: [/var/lib/containers/storage/overlay/5ac6da02d68115a4725293d311ddbbc70f17cd2de2e8c541401f55b429f43bba/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.284864    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880420025s: [/var/lib/containers/storage/overlay/f3497cc283a2c4a6edca9c59f8cfc17eebc8bf3a209baef0ce50e4022f2e5b87/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285034    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880587372s: [/var/lib/containers/storage/overlay/64eec43892d3dd9169458d3fcdcbeac18b9c7d0c788b0032bf03db2b398fb897/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285060    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880612222s: [/var/lib/containers/storage/overlay/68be65f5cc54c77cc9038e4c4b860e5dd1e5f43fbc5ec45bd2f360df3876e30e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285084    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880635282s: [/var/lib/containers/storage/overlay/fdbad914a75a4662f51ea3de79e3b5fa40a885c1e10084436c7952c883c5b47b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285110    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880658895s: [/var/lib/containers/storage/overlay/49c4dec32539f7025afcc2528c430b54c264f370cf53ba0cbb14bb6da106dd0c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285137    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880675179s: [/var/lib/containers/storage/overlay/52867d1e7d05271ca47676eedec04dcc21f81ee80b9b47038d1a8a8081ba3049/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285169    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880707176s: [/var/lib/containers/storage/overlay/de6fa16e2c92d7607e2dfa734637d7a358d08601c4841a2c70b9bdc70451d2a2/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285188    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.436578894s: [/var/lib/containers/storage/overlay/91d4a8b25c28866a02290e6bb18165a4d76ac797eb0589d2b20e4527357d3806/diff /var/log/pods/openshift-ingress_e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk_e43f7199-d45f-4e71-b5dc-7bd24a975284/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285205    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.295781614s: [/var/lib/containers/storage/overlay/706bfb24efc8a0c4b7a31165cc6b9c357491920a2ca95ca92908f92fa645328b/diff /var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-597546666c-728zv_0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285218    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.072833805s: [/var/lib/containers/storage/overlay/70053a80acf68decd8d2350c5cc6f42c602f77169eb855d1e45f254aa62a6894/diff /var/log/pods/openshift-ingress_e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr_e7af977b-c72f-4da5-81b1-e5eb0de7ead0/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285231    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.055195531s: [/var/lib/containers/storage/overlay/ff85abaf85cd27e7c96e2500f9f4416933957918ae68fa7eea2cb0dfca9805ef/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285244    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.055202809s: [/var/lib/containers/storage/overlay/bc854e9c3156e2247420079d038d67a374b22eca079b4ade70a32b0723ffdf11/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285491    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285528    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285569    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.285588    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.303187    5797 trace.go:236] Trace[509823243]: "Calculate volume metrics of tmp-dir for pod e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" (04-Nov-2026 01:24:48.041) (total time: 4751ms):
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: Trace[509823243]: [4.75176156s] [4.75176156s] END
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.303597    5797 trace.go:236] Trace[2092685818]: "Calculate volume metrics of konnectivity-ca for pod kube-system/konnectivity-agent-6ch6l" (04-Nov-2026 01:24:51.966) (total time: 1336ms):
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: Trace[2092685818]: [1.336952845s] [1.336952845s] END
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.378563    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.378668    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.378697    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.387960    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:03.588944 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:53.388113    5797 logging.go:55] [core] [Channel #1431 SubChannel #1444]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: operation was canceled"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.390073    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.46:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.390367    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:53.390467    5797 logging.go:55] [core] [Channel #1441 SubChannel #1445]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.45:50052: operation was canceled"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.392051    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.162021646s: [/var/lib/containers/storage/overlay/c70ecaf40642e2fde7f3cb40b265cf03ff440495b00f26048670d5bf3d744674/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.392354    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.934771678s: [/var/lib/containers/storage/overlay/c972520930be744cfe427c9b7e62f3b0974a298d9b54f145d0ab502a481084be/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.392649    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935059094s: [/var/lib/containers/storage/overlay/1d292cdca831efe9e0f234ffe407a63ddadfabd2150b081db3450f491f186864/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.392809    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935217159s: [/var/lib/containers/storage/overlay/3207ad0951fde6a9deb7038c93ae401caeb5baba58b31e0e26dfe47027a5b61c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.392836    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935243051s: [/var/lib/containers/storage/overlay/6ea81b2c5023f8dff946e71b84f945b9da2b10fcf1258a35cb8a51bc0c32ca32/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.392867    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935272674s: [/var/lib/containers/storage/overlay/c877290bd37df4d0b881b5c91502b3367fa267d1cdf1824550823add4eae7cad/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.393074    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935476917s: [/var/lib/containers/storage/overlay/84168eb603554221b794afad63f68288dbd8be1b47e6a73d7313bc77b4e33e59/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.393104    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935506068s: [/var/lib/containers/storage/overlay/aa0095d066463967054f3b423673856a9bbec70334c38298460ed75b838a6edc/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.393271    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.935670247s: [/var/lib/containers/storage/overlay/79ebd463e4f596c283513cb027f031ade0f9f09d62dbf627d004ced716a26579/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.393852    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.936241985s: [/var/lib/containers/storage/overlay/6646ae3e529a6a19d3fdf96e60eef307a1185af40970672e9fd5e0fbd0fc8f97/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.394051    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.936439267s: [/var/lib/containers/storage/overlay/10da7338e6963a217024b6491a878d03979b2b226a0e0aaef56fc95fbaa425fc/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.399295    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.399922    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.16985835s: [/var/lib/containers/storage/overlay/5c7baa8769be39a6e9e4da944d1d7de17e02e7080ed078dc2310e0f6b0f7243e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400014    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.942095387s: [/var/lib/containers/storage/overlay/d7a0cf31d3468144605843042b1605c8d729324b6b22fe69de5b3cfa34f7c223/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400053    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925348243s: [/var/lib/containers/storage/overlay/ae26cfe3298f50018624be0996602265f351907caa9395363d190007bd776760/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400079    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925342748s: [/var/lib/containers/storage/overlay/f651598829209a878c97e688d52a21516151b27299622634f81f1654afabe891/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400108    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925373175s: [/var/lib/containers/storage/overlay/ef4ccd615931a8e970ba4784bfbda50602d3e4062a6787d3a826c633c1c75d4f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:53.400137    5797 logging.go:55] [core] [Channel #1430 SubChannel #1446]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:50052", ServerName: "10.134.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:50052: operation was canceled"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400139    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925400474s: [/var/lib/containers/storage/overlay/e5e0eca001df26036c6ce8021d914fdd2cc103ed4af17aeb8423f77e10fc06e5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400153    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925417204s: [/var/lib/containers/storage/overlay/225eee4db49dc38b24f446fd78d42a140ce87155fd32665c92d52a92af2acd53/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400163    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925425276s: [/var/lib/containers/storage/overlay/0ddcea9afa6ed31457807ae8d1819c6ac1c6579d308ecfbf624699d41f015a2c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400187    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925446861s: [/var/lib/containers/storage/overlay/4b851b7ec3e16daa8033667bc838217557ab1f5af2c0387942bf966e16dee50c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400211    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.92547009s: [/var/lib/containers/storage/overlay/669d78fe536799f67894bdf99fe62922cddac59e73ee41fcec2d3548c14d3b31/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400207    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjqs2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400237    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400254    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925509489s: [/var/lib/containers/storage/overlay/c33750a2beae957e7a117d60282dd804e856e96ae2d6201927413075218bf364/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400296    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925550621s: [/var/lib/containers/storage/overlay/3e5a94662e98ef753663170b3a2fb2afc2cbf84155169aa3bd2f20155b541ba8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400336    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925584659s: [/var/lib/containers/storage/overlay/a0359ad7995b3a551057a39f9e98b6d4e5d578ddf8545a873c434f788fb39aa9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400363    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925607225s: [/var/lib/containers/storage/overlay/cd434610b4ffd4e25a8e59f8c3c615f91c24799e3be2855969265e9ccf57a99f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400373    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.9256202s: [/var/lib/containers/storage/overlay/540552188d0a86126965d7261fa7bfec2cd2a34efebeafdb19605bb535dcae55/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400410    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925136326s: [/var/lib/containers/storage/overlay/a5c0be588a2adb129a9dd30f70f023fe2d32d4d07375a1b96734d986d231d651/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400446    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925172182s: [/var/lib/containers/storage/overlay/7cf1c06cd90ea7b850c027f5cc1d2a1a0d72a9ebd384a1055b12e6d2cc4bb199/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400485    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.92520968s: [/var/lib/containers/storage/overlay/f03d67d1ff00f3567394819b553560a90344872a14f25ba12417bf1fcca8b7bf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400523    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925246098s: [/var/lib/containers/storage/overlay/5c8ef5655ac0b6b14ef7d4180ebfe620a8f1c3bbac8add11688285a3271c9405/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400559    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925280207s: [/var/lib/containers/storage/overlay/807ea018e934c59eae2ddc044bb1b27b7ae4e0cd26987f223aa18bbe058f2d50/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400599    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925318557s: [/var/lib/containers/storage/overlay/44e79a45373cebc942d8f41ba36aaeb883656bd9cd3f0bb8cb9cce8593b15be0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.400638    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925353982s: [/var/lib/containers/storage/overlay/51af17e2bcfee40fe26a1eb6b4ebbbd8dafe77fd6a250ca8ecd45d7d13963c16/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.401029    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925736419s: [/var/lib/containers/storage/overlay/f9a9ac7e5e6a75a93dde3689ba0115d0feca1c05e0f94ed99f0b36f3ddac20be/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.401071    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.925776728s: [/var/lib/containers/storage/overlay/cebb62c07b8997298ed9746e094dd43bf0aff3bcb4db113b72b1a7aa5158f541/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.401087    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.171048958s: [/var/lib/containers/storage/overlay/e5a9c7b1fe7001b2823f683ffc4b4241fa08d8f54d047a1be541de98768e7f0c/diff /var/log/pods/openshift-ingress_e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd_a00f3b41-2394-44ed-9802-cb14cb74f785/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.401484    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:53.401578    5797 logging.go:55] [core] [Channel #1435 SubChannel #1447]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.49:50052: operation was canceled"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.402736    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.172714099s: [/var/lib/containers/storage/overlay/a69741fe4b6a660208036fa19f11d3ed9ba39625be19ab5968018d1503ad968a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.402778    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.290047839s: [/var/lib/containers/storage/overlay/6e31770ff1045be85439e600dba30e9e5d163423f258e0998f9a623732464e74/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.404056    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.288805031s: [/var/lib/containers/storage/overlay/b17b6293ce69ec07e1a34eedb1862b162e500b5d5b39b743f746a65b99f0db46/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.405171    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.17514225s: [/var/lib/containers/storage/overlay/014d3bbf6e7cba35dce964f331493aae4156e3d4c27d5a869c1bb2b9f09733bf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.405199    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.199613289s: [/var/lib/containers/storage/overlay/eeefa8873ce600b2aa1be580549e32d507efad39d752bac4130ae61d6e671826/diff /var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-gsw2f_35506939-7454-4450-9d9b-687b5caaea77/postgres/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.405546    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.405575    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:24:53.406002    5797 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:24:53.406025    5797 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:24:53.406046    5797 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:24:53.406059    5797 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.407510    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.409419    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.419377    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.419646    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.419677    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.419719    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.420900    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.420930    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.421029    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.421050    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.421098    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.421117    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.421879    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:53.422145    5797 logging.go:55] [core] [Channel #1436 SubChannel #1439]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.47:50052", ServerName: "10.134.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.47:50052: operation was canceled"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.437351    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.437386    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.443773    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:50052\" within 1s: context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:53.443904    5797 logging.go:55] [core] [Channel #1443 SubChannel #1449]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:50052", ServerName: "10.134.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.48:50052: operation was canceled"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.444592    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.444620    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.444722    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.444751    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.450350    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.450398    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.450522    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.220455083s: [/var/lib/containers/storage/overlay/1f599880e321736428ab06be2a6f4494122464ad1f718105f72164355d51dbcf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.450584    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 11.900200265s: [/var/lib/containers/storage/overlay/99b116f0409a4025870a09b6e86de4e281653aa17d8b70f2bc8e2eacec76b6cd/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.451639    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.451675    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.454701    5797 trace.go:236] Trace[1032196341]: "Calculate volume metrics of secret-telemeter-client for pod openshift-monitoring/telemeter-client-b6c7df88-lkrmc" (04-Nov-2026 01:24:37.419) (total time: 16035ms):
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: Trace[1032196341]: [16.035042539s] [16.035042539s] END
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.479412    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.479978    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.480225    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.490818    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.833469    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.833542    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.850844    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.852386    5797 trace.go:236] Trace[513294909]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" (04-Nov-2026 01:24:52.178) (total time: 1673ms):
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: Trace[513294909]: [1.673974212s] [1.673974212s] END
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.852798    5797 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-597546666c-728zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.852843    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podUID="0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.852868    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.203181771s: [/var/lib/containers/storage/overlay/e85d2eff541ba57a56d0c53fad694e2b2ba539c45fadca450e1c11a760cd5c93/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:24:53.866380    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="e388e6ecb155bb8f242c995eceb4ebe2f48d1411583234e4295dc5a8e968d2a3"
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.978748    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.801963246s: [/var/lib/containers/storage/overlay/8b5418186ab26ba9508bc3879927ffc8956e9854bc382de94ceb2f8398b9bef3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981086    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.751050798s: [/var/lib/containers/storage/overlay/1cd65d6222cf2a2f228b0eeb5fc482d7b154616e229b436fc304a09bdd6d5ea9/diff /var/log/pods/openshift-dns_dns-default-z88mh_b1de55b1-797e-4e61-aeee-624aa4e20ca5/dns/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981148    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.751078106s: [/var/lib/containers/storage/overlay/5079d183db069d19cc411904c0cf55540ba3fdaa8716ac969338cb8edd62b6f0/diff /var/log/pods/openshift-multus_network-metrics-daemon-gfjzp_4b6e932c-bb1d-424e-8dd2-c92797151de6/kube-rbac-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981191    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.751217552s: [/var/lib/containers/storage/overlay/ac5f9f523fc152a7248e963e6043df0546e0b04a67311eaaea9a99b4ec495202/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bw7dk_9693b4e7-8f19-4e24-aac8-4fb293180bae/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981445    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.751403953s: [/var/lib/containers/storage/overlay/2a5a0a2ff56f62b4eaef94e42d0520129b84675ba0b18a3351df355e54d93897/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/northd/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981412    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.75134966s: [/var/lib/containers/storage/overlay/f6a18b141f7f8bdcd7b330b96fe42642a4aa8fe6787735748cf33d42dc74ebee/diff /var/log/pods/openshift-multus_network-metrics-daemon-gfjzp_4b6e932c-bb1d-424e-8dd2-c92797151de6/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981488    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.749688323s: [/var/lib/containers/storage/overlay/b0b618070ba355a83f65b255e8badeb00076abbaa051ee70dada4c0a69f9e37f/diff /var/log/pods/e2e-models-e2e-worker-w1-1036e8_premium-w1-1036e8-kserve-c94f848c8-wr4tt_c70573f5-5de4-4121-a727-c65799a4da86/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981518    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.577117888s: [/var/lib/containers/storage/overlay/6d4de21fe385ee537005a864c5f47d8cf5562ae06373709ecb766559adc62cbf/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981555    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.577154455s: [/var/lib/containers/storage/overlay/e711e17a335eab10a4488ae8734d72db6a0e5e07ebac488efbc052a1e6c88ab6/diff /var/log/pods/openshift-monitoring_metrics-server-5bb86d5d94-r4w4m_1a2e54ab-acb8-4611-bf2d-534053133f77/metrics-server/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981562    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.577138797s: [/var/lib/containers/storage/overlay/6ec4d5b9d784ddf0bd2d6fe3b5f5bbccce5f4dcf15d9c6745fca4ba227880f4f/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bw7dk_9693b4e7-8f19-4e24-aac8-4fb293180bae/kube-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981573    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.577135445s: [/var/lib/containers/storage/overlay/ed31867815e48f1055bb9b4926dc2b1182cec9dacd7ab3f0a4df77df063083f9/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981597    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.577149963s: [/var/lib/containers/storage/overlay/9e9fa44747f445ed586ce9e7d3cbccfcfde626917fa8197cef4155d546e2548b/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-186.us-east-2.compute.internal_8fe1c320288f5f6b9ae9ef95fa346d10/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981597    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.577223385s: [/var/lib/containers/storage/overlay/301060f3578065870868d3b40fe325c71d2432fd0508b3b9b89ff2f16b5b69bd/diff /var/log/pods/kuadrant-system_authorino-ff97f47cd-jh7sr_9f7141ac-7569-418e-9880-8f863b8211d4/authorino/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981620    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.749656558s: [/var/lib/containers/storage/overlay/8d8106be1b05460ee141774b805e12238737332f7cb65366fd0beb2445baeb4f/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bjqs2_f6f7bd07-f7a6-4817-9644-63ceeff7a207/discovery/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981638    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524059819s: [/var/lib/containers/storage/overlay/87631b79d11d987e932877dee2a8c174dd57ab9c0260fc3d420903a3abe3e36d/diff /var/log/pods/openshift-dns_node-resolver-6tjgh_95384914-9263-45e9-9639-3566767d7559/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981664    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524084714s: [/var/lib/containers/storage/overlay/92c9ccad9adbe31767c9466d8a94d4b46b288870e444978794df6369c4ae1427/diff /var/log/pods/kube-system_global-pull-secret-syncer-x7c8v_bc5258b8-2791-49ac-a271-e914db291c6f/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981681    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524100204s: [/var/lib/containers/storage/overlay/38808130ed3d3ebccea5c10f83b8523d180cf4bf0a98d5439342ac8fc1f4e9a8/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal_388b6cc67d09cb3e8a1219d1e84089fb/haproxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981696    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524113744s: [/var/lib/containers/storage/overlay/7a3ea1fd0852076e3d481e9e77d884880d10dac96db00091e22916b69b44d19b/diff /var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981714    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.52412414s: [/var/lib/containers/storage/overlay/abc69d605733a495ef6a5ec69e9b844f298d4256fdb34b6e0b8698a903e6340c/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981735    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524139745s: [/var/lib/containers/storage/overlay/f2dffdd23a54d02ef3aefdafb1ba8828af162936dcf5492253c9ded7b2386def/diff /var/log/pods/openshift-monitoring_telemeter-client-b6c7df88-lkrmc_77b967b9-f2b0-4b23-9484-038568e3a0ea/telemeter-client/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981759    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524160097s: [/var/lib/containers/storage/overlay/4cabdf59a5b21f04146e37714dc5ecdac418a441d96c59bcc40155c6f6a2c4c3/diff /var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dxwgn_b5df0493-dbfc-4398-94fa-2d5ec7676197/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981774    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524173588s: [/var/lib/containers/storage/overlay/af97afa97d0b66dac5bc6c7b0b818117f7885bc204289be5ae032d0af77f9215/diff /var/log/pods/openshift-insights_insights-runtime-extractor-lbqr9_bf31dccb-516a-4a97-aa86-9277ec17154d/extractor/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981790    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524187134s: [/var/lib/containers/storage/overlay/631feb1d7b09c778c1ad76a8603579bbd7b2b5e496d29d63b87b417c703682f7/diff /var/log/pods/openshift-multus_multus-jxdjh_e8fac715-d046-4c8f-af92-66684a66e0ae/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981815    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524204831s: [/var/lib/containers/storage/overlay/101d1f971f73c7d569e4a56adec65d9f2ce6ac3cb490a905e4ceda60ab8dd400/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981830    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524218021s: [/var/lib/containers/storage/overlay/521bf0cc8f0bbc64e500485ac7cb7920433a9fea2de685b088bb75bcc79d90c9/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-vf66l_552f596f-96f2-4d27-9b5a-1c87f8dc864d/tuned/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981847    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.751877084s: [/var/lib/containers/storage/overlay/a841f36a6963463c906d1ec3f4d47e9b904ae0c87a976dc45d73448d8496dc45/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981873    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524254533s: [/var/lib/containers/storage/overlay/85b1ed019f7fa1d945aaa9c06e130ca6149a8a65e8b4f426712edba6b708d829/diff /var/log/pods/openshift-dns_dns-default-z88mh_b1de55b1-797e-4e61-aeee-624aa4e20ca5/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981893    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.507153965s: [/var/lib/containers/storage/overlay/9933e53656b979766254f346ed32d97d05cb6328e0f0f4b76c62ef06727f88c7/diff /var/log/pods/openshift-insights_insights-runtime-extractor-lbqr9_bf31dccb-516a-4a97-aa86-9277ec17154d/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981916    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.506644612s: [/var/lib/containers/storage/overlay/368c27d56fe1d072d6a80a1ca2ca5696061606b84b5c557f0bf905c801a74c95/diff /var/log/pods/openshift-ingress-canary_ingress-canary-z5jfh_f132805a-066a-4d03-bcb3-90c1dc9dc73e/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981946    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.506663734s: [/var/lib/containers/storage/overlay/a3cf4d236ee4249e2632ca7d787a35a74aafca0288c2b7e5c8913438b25c7450/diff /var/log/pods/openshift-network-operator_iptables-alerter-x7xvv_33e0d88d-b649-469b-905b-25cded0252f9/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981966    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.50668239s: [/var/lib/containers/storage/overlay/72dd28b52ef4d8ff9d4c7dbf602c4db2a36f168046f0cf82aab6505471654e97/diff /var/log/pods/openshift-insights_insights-runtime-extractor-lbqr9_bf31dccb-516a-4a97-aa86-9277ec17154d/exporter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.981981    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.506703473s: [/var/lib/containers/storage/overlay/367afcf747fa127c232d5a48da66131bce7e6e1127e37c27c51b2fca9749c09a/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982018    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.506727459s: [/var/lib/containers/storage/overlay/f03bc44a84f68d20061838f5a87309e0866d308272e899118d333b69ab88e6db/diff /var/log/pods/openshift-monitoring_telemeter-client-b6c7df88-lkrmc_77b967b9-f2b0-4b23-9484-038568e3a0ea/reload/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982042    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.868569239s: [/var/lib/containers/storage/overlay/5e76d41b1781bc00ff1b977a5c760879500b8a78274aae752b24bf195a5c5753/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-fjpq6_bd730006-869f-4fb3-b626-912e7530f33c/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982062    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.752040435s: [/var/lib/containers/storage/overlay/07fe0a1f131428f9c8b7cec998307a67b28a3024ee1f089d84c121fc10ac862b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5fq7t_9e38076f-6754-46f2-b054-f0aa1976319e/sbdb/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982080    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.752050722s: [/var/lib/containers/storage/overlay/a90505cd6ff459369fd0aaed98c93e87012c211800d7d66f089d47b056569c34/diff /var/log/pods/openshift-monitoring_node-exporter-29mzn_513cdfcf-ac71-4d24-aca9-1c81edb6f803/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982094    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.752063068s: [/var/lib/containers/storage/overlay/dbcb067d307229bcf31111c1c82399b6d75415dcaa67df3f7080003eae8ff81a/diff /var/log/pods/openshift-monitoring_telemeter-client-b6c7df88-lkrmc_77b967b9-f2b0-4b23-9484-038568e3a0ea/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982115    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.84197988s: [/var/lib/containers/storage/overlay/2f681086a5a9e86fb1404e3aa4f2b159d1596766775cd320f2cf10515660cadb/diff /var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-7nshw_e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7/jobset-operator/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982137    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.868266388s: [/var/lib/containers/storage/overlay/cbc5a81defbb786831adf80778394b14a6bc65ab13462a16d68e9b8799f44652/diff /var/log/pods/openshift-ingress_e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq_5d17dd83-848f-480e-b510-3580fab17a1a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982173    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.75393443s: [/var/lib/containers/storage/overlay/9370d6984a7b30de390e36f5feff56b9161d3ee13ac080b08413c992f88eaaf5/diff /var/log/pods/e2e-models-e2e-worker-w5-875528_simulator-w5-875528-kserve-7bbfcbdb95-cj9tv_2a407479-18d4-4b04-ba29-7961fcf5f538/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:25:52.532299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982195    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.753561422s: [/var/lib/containers/storage/overlay/20726bbda89779629ae346fda50aa3c7b9ec06583f485666d8005958f11a353d/diff /var/log/pods/e2e-models-e2e-worker-w5-875528_distinct-w5-875528-kserve-77c8c8df7-brdl2_d8c17db7-cbad-41ff-92f4-58bc2d4a6458/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982154    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.52604307s: [/var/lib/containers/storage/overlay/42881845c7d8cfe41f411b7162329b9ba3414d830a1da20a04d2eb684987ce32/diff /var/log/pods/e2e-models-e2e-worker-w1-1036e8_distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk_29348c5f-115b-4823-9a13-fe0c34d6765d/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982213    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.375239991s: [/var/lib/containers/storage/overlay/d252df8b15413d6213786ecc098762c0fc2907b47a07ee8e35f9f154cc8c9cf9/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qb7m9_270dc588-8548-488f-a219-e745a8d110db/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982230    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.239829283s: [/var/lib/containers/storage/overlay/fdf7a2c4a499f60445fbd46d2b7c101519831b642d13b3c52dfe6424b635b808/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-ffngz_69fdf072-b1d9-4eca-83dd-fc22311d4a32/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982251    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.869530263s: [/var/lib/containers/storage/overlay/9ac8e42cf5650f8917715c75f0275e4a073693c692081a7eb2b938406775753b/diff /var/log/pods/openshift-ingress_e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9_262bf953-f280-410a-b214-67b2af0cd83c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982266    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 10.680291812s: [/var/lib/containers/storage/overlay/a3f888fcd91484a423fdda8bb17ae171bcfa0bd588d7a4be528da776dd09f1f6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m_e4ef37fa-243b-4c4f-a707-bf1b05287a42/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982297    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524940458s: [/var/lib/containers/storage/overlay/2d6ba150daf8c3415ea80ea59a44afd41630f4a56d0e44d7338885aa64d809c9/diff /var/log/pods/openshift-ingress_e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq_94d933d6-8278-432e-bdf4-7bbd5d63f8b7/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982349    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.524730124s: [/var/lib/containers/storage/overlay/19ff757bdec0d0a031a2c5178ce1abe5c25aea9568c4730af48965240975851d/diff /var/log/pods/openshift-image-registry_node-ca-dnjzj_22778bfc-2110-48e0-85dd-25e5ca26c161/node-ca/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982385    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.849663794s: [/var/lib/containers/storage/overlay/a1b148dcc870c82286b084f98c960120fce6b2e9ee94bfa85c8237e16a7d4ba2/diff /var/log/pods/openshift-ingress_e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9_495c9d92-defe-4156-9c45-8e6ed42e7b5b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982426    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 15.718394057s: [/var/lib/containers/storage/overlay/ee3e9fea0e17b1aaed3d0c1182fe7a794aefac92f57c012bf090e6b97ab7afcf/diff /var/log/pods/openshift-ingress_e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd_d6f64df9-087e-45ed-b182-50ce24b355ba/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982449    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.52570314s: [/var/lib/containers/storage/overlay/d8b856406a81262be8b00353a7785005cb44dc7b245fb4196e8781d388724a24/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8ff7fd4ff-hxltx_3dba5003-a608-45f0-bee9-a0597e2d39b4/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982472    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.753918408s: [/var/lib/containers/storage/overlay/db6ad60bec42456505872b6dbe418bf302fbcc2869efe2ff31af47897943dae9/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-b44x4_fd4a4e92-a390-4fc8-813d-71dff7e4e07c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982511    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.578111656s: [/var/lib/containers/storage/overlay/915bd673b3b10c5c546a225d814d02e35f62e931a340602c2ef26b95796d8515/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-lvdcj_6b6ded4e-6d49-4cda-bbb9-243a999d05e9/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982530    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.428088745s: [/var/lib/containers/storage/overlay/30b95860137b8a68ba43b076a58c9569c22d3ba1c0448710b4211b9e743efeba/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm_4fb22100-7a0e-49ee-9ab4-66b2a54c34c8/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982554    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 10.646630869s: [/var/lib/containers/storage/overlay/6f0ecc0e8e1a0796130416cb8d8b895334e8a0e0d8757b9c836d4291f5e2d6eb/diff /var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-654z6_517f1a3c-9ae6-4bbb-a412-22101ed0784f/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982575    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.861596353s: [/var/lib/containers/storage/overlay/863d00a5b124e71826549b06a5faa3df51d433d8207480f84c2b2f140f01cc5c/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-dw6nw_fe939348-5f96-41ac-b733-2746eb79904d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982603    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.99021096s: [/var/lib/containers/storage/overlay/8df262f0d47790998850a38d29d99c530c396a845778694a040f384becc36d7b/diff /var/log/pods/openshift-console_console-6b8dc999f8-b5bn9_f51cfd3b-b1ee-4361-aef7-d80d21dac86a/console/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982632    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.843932085s: [/var/lib/containers/storage/overlay/43df93443e9bca26bb2344665a9365f780d437e176305401dd44376d3d2d697e/diff /var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mvf9p_6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982667    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.941634412s: [/var/lib/containers/storage/overlay/c84240acad8a8d2d8da49cf3b841a70322020b917e42ebd2c91283ad3e8c571c/diff /var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7l6kd_f493fdd3-b7e8-493f-af2e-df12f9651df0/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982665    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 9.545897153s: [/var/lib/containers/storage/overlay/87142ba260b28d5767e1ff2950d09d545fecf2fba75eb08d6a6d05f8cf00a4f2/diff /var/log/pods/openshift-ingress_e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj_af20f3c0-5693-4ff4-9587-616247f7553e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982697    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.941666032s: [/var/lib/containers/storage/overlay/0b2e22d2e71eb84831941ffc766a70c0e0a1dad4d9314bc5bc9b15e9fda6265c/diff /var/log/pods/openshift-ingress_e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk_b3056bef-bc61-4c70-8aa9-1a0673bf5733/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982713    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.862189945s: [/var/lib/containers/storage/overlay/8509f4575186f4fdef35a1a8dbe36eec4548827088b033ee823760f9f82a097c/diff /var/log/pods/openshift-ingress_e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2_0a85dd86-0f39-49f1-ac8e-0986ee8962b8/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.982741    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.941712307s: [/var/lib/containers/storage/overlay/e089e3cd3a0a0a6bfb81c262eea5ad3ae242330a19eb7a7e8d44b61ec5c0e0fd/diff /var/log/pods/openshift-ingress_e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9_f4f6a452-5b44-4a9c-b3d3-f7508f95fa38/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.989026    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.814334514s: [/var/lib/containers/storage/overlay/e388e3af01cd34e5e3717d276dc59de9dadb152d79732f451f4cb67c6f87a453/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-775c47d899-9l8hb_bd717794-dd18-4bb7-8dfe-222545f12ff7/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.989656    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.340755779s: [/var/lib/containers/storage/overlay/671f7994bebef91e2354e44cd4f46b294c1bcaf04c35acdbafec34ac4016befe/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld_148e1b22-7bb1-4399-950c-66a0608a202a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:53.992273    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.145606979s: [/var/lib/containers/storage/overlay/bbffa3a674d1cb128ac67012ec0f5f511ddabbd036e35d24a54c19d1956d4e1e/diff /var/log/pods/openshift-ingress_e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2_9c7f34d0-b8a1-418f-895b-3a5b61c57bda/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.039141    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.413023146s: [/var/lib/containers/storage/overlay/56ffe7881c2ddcc2fd170bf4698aaefe449977bdaada0790a1a0b22959ada749/diff /var/log/pods/openshift-ingress_e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn_66bc8ba2-e063-4d0e-b12d-7a73718366c7/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.047937    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.981212187s: [/var/lib/containers/storage/overlay/d1325e0cd0f5911385a450f059814291f775618d49e0a3ce4c4df3cf8a19e82e/diff /var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-smsgb_b96933e9-892e-41b6-9b0f-9568cb44c538/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.070341    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.949808515s: [/var/lib/containers/storage/overlay/09632907d873387f4d7d2c38617de07a2391223dbd42223fbf9dbaa3b022dbcf/diff /var/log/pods/opendatahub_odh-model-controller-769875db56-jr85b_8931f2f8-db63-47c5-9e05-15b4a0eb9c51/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.071456    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.004733835s: [/var/lib/containers/storage/overlay/29e3020132f475cf4b4ee12a5efcd16a6c0c9e79644e83b93641663dd5f029b2/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck_a2383056-569f-4805-8ee0-23ee3268483a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.073739    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.10633047s: [/var/lib/containers/storage/overlay/0109308a46f75f95632afe8f46e5ad33f2cf3e11852f11adc656f9a070bdec9e/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-x2k4g_c6b2d0c8-f594-4dc9-b8ff-a3dec860c112/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.078106    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.849425877s: [/var/lib/containers/storage/overlay/bc5e68f2d1ea2cccb0424cb52e8a72dc8bc64906e936e82e2d7c2d2256a20174/diff /var/log/pods/openshift-ingress_e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr_0da19693-73e3-4bb8-9637-c68ec381334e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.078984    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.01227431s: [/var/lib/containers/storage/overlay/b8b78e49f4a4216ebe95416ff35795a452130f5bcaeb67856ba1feecf8ea45fa/diff /var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-pqxmd_5d133c42-6f8e-420e-861b-37d8e673161b/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.109737    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.195880539s: [/var/lib/containers/storage/overlay/874dad3110da4a0295b0f0ae3dfcf0cea95a9c780bdfaad16178a8a8fc44a0c7/diff /var/log/pods/openshift-ingress_e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd_3fa7e470-2922-43cd-a478-714d01ed9e64/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.148711    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.170250    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.883845038s: [/var/lib/containers/storage/overlay/4b94f95c3e0f2f2414b3fd645fb092e11e748759a7a8400c47dfd380104488c4/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh_fb840530-c49b-4bad-b7b7-d94b4c7dd0c8/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.190146    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.192041    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.070568866s: [/var/lib/containers/storage/overlay/73d639c4b7926332df28bc7c875756c36aa628e768d7bf845db03d11f244d0f3/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd_c78d06ce-2464-4370-928b-3935d9667f66/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.192134    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.905695196s: [/var/lib/containers/storage/overlay/9bb6a03552796a9981ebe54864b42e17aabb1caf9c53880f662261382474f9b2/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9_dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.192134    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.192188    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.200103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.257123    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.258169    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.258224    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.267110    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.267170    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.269078    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.269128    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.270088    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.271101    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.271150    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.272204    5797 trace.go:236] Trace[968362027]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" (04-Nov-2026 01:24:53.223) (total time: 1048ms):
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: Trace[968362027]: [1.048689572s] [1.048689572s] END
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.273452    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.206742901s: [/var/lib/containers/storage/overlay/6c37e605caa56fcc7f5a52a5ec31d03454c7fda6f973ed26c0510a601eb94653/diff /var/log/pods/e2e-models-e2e-worker-w3-a5115a_premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc_c958dc33-8c5d-41ae-8156-41ef575354cf/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.275122    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.275168    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284094    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284108    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284139    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284098    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284156    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284169    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284183    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.284141    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.292091    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.292128    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.292089    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.292168    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.303106    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.303145    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.410920    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.410979    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.417779    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.438105    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.453100    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.453143    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.454123    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.482136    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.482154    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.482185    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.485092    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.485130    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.686688    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.45708293s: [/var/lib/containers/storage/overlay/47a74bc3f559dd85cdf02e276a158f34b6f36dee2dbf9c9cb894907115339d6f/diff /var/log/pods/openshift-ingress_e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2_5f9ad66a-bede-4e9a-94ea-f9dd9a924f61/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.706963    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.479202456s: [/var/lib/containers/storage/overlay/f29411d195036a0d71cef4f106ebd48c085a88296caca9676da80f1ba39d047f/diff /var/log/pods/openshift-ingress_e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz_33f3f13f-1942-4dca-b993-0f48deaa503e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.860147    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.46:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.865953    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.866153    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.866201    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.866222    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.866876    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.867168    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.867336    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.867474    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.868503    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.869281    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.871920    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.878230    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.878400    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.878471    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.885277    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.885288    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.900133    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.900190    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.901103    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.901148    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.902139    5797 logging.go:55] [core] [Channel #1451 SubChannel #1467]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.46:50052", ServerName: "10.134.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58764->10.134.0.46:50052: use of closed network connection"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.907126    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.907139    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.907185    5797 logging.go:55] [core] [Channel #1450 SubChannel #1462]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56368->10.134.0.50:50052: use of closed network connection"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.907189    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.907215    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.907275    5797 logging.go:55] [core] [Channel #1452 SubChannel #1469]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:50202->10.134.0.45:50052: use of closed network connection"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.907333    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.908162    5797 logging.go:55] [core] [Channel #1454 SubChannel #1459]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46958->10.134.0.49:50052: use of closed network connection"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.909114    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.909143    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.909159    5797 logging.go:55] [core] [Channel #1453 SubChannel #1457]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:50052", ServerName: "10.134.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46942->10.134.0.51:50052: use of closed network connection"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.909173    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.909184    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.909205    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910101    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910148    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910232    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910180    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910302    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:15.869245 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.910143    5797 logging.go:55] [core] [Channel #1464 SubChannel #1465]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:50052", ServerName: "10.134.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:48360->10.134.0.48:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910138    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910352    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910179    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910466    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910183    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910582    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:54.910241    5797 logging.go:55] [core] [Channel #1455 SubChannel #1456]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.47:50052", ServerName: "10.134.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:57580->10.134.0.47:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910118    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910649    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910837    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.910864    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.911104    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.911142    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.912130    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:54.912158    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.138113    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.138182    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.241146    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.258224    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.259350    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.259390    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.267468    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.267521    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.269601    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.269653    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.272084    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.272120    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.276249    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.276288    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.292284    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.292330    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.292400    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.292435    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.300730    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.303423    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.303502    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.412053    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.412127    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.439416    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.453493    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.453547    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.454536    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.482821    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.482844    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.482885    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.486123    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.486161    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.866711    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.866958    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.46:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:55.867020    5797 logging.go:55] [core] [Channel #1473 SubChannel #1474]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56376->10.134.0.50:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.866878    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:55.867121    5797 logging.go:55] [core] [Channel #1471 SubChannel #1472]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.46:50052", ServerName: "10.134.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58780->10.134.0.46:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.867172    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.867530    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:55.867602    5797 logging.go:55] [core] [Channel #1476 SubChannel #1479]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:50052", ServerName: "10.134.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46952->10.134.0.51:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.867658    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:55.867753    5797 logging.go:55] [core] [Channel #1477 SubChannel #1480]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46984->10.134.0.49:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.869438    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:24:55.869575    5797 logging.go:55] [core] [Channel #1483 SubChannel #1484]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.47:50052", ServerName: "10.134.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:57590->10.134.0.47:50052: use of closed network connection"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.872487    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.879064    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.879070    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.886326    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.900589    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.900632    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.907084    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.907129    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.909249    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.909328    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.909363    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.910710    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.910753    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.910838    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.910859    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.910924    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.911278    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.911314    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.911383    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.911398    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.911457    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:55.911471    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:56.133985    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:56.134099    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:24:56.444341    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:49.861656    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjqs2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:49.861741    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.102200    5797 trace.go:236] Trace[483456500]: "iptables ChainExists" (04-Nov-2026 01:24:35.229) (total time: 74872ms):
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: Trace[483456500]: [1m14.872193504s] [1m14.872193504s] END
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.102257    5797 trace.go:236] Trace[1583347123]: "iptables ChainExists" (04-Nov-2026 01:24:36.113) (total time: 73988ms):
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: Trace[1583347123]: [1m13.988781989s] [1m13.988781989s] END
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.221794    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.221870    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.225143    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.225193    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.239562    5797 patch_prober.go:29] interesting pod/network-check-target-58tg6 container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.3:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.239611    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-58tg6" podUID="b059c4d3-70d1-4c04-95e9-e1735da565ef" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.3:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.241519    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.245803    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.245842    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.247552    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.781009488s: [/var/lib/containers/storage/overlay/22e8480c40187870a896ffd09ccd904b226bfe2fa05e0729c980c39be3b2cf36/diff /var/log/pods/openshift-ingress_e2e-mt-e7742169-openshift-default-6986f46949-dgng6_59912ca4-fdd9-4f5b-9713-220f3136e515/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250145    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250182    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250373    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250396    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250535    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250559    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250682    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250702    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250836    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250857    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.250978    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-6ch6l" podUID="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.251036    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-6ch6l" podUID="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.251455    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.251481    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.251641    5797 patch_prober.go:29] interesting pod/dns-default-z88mh container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.251660    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-z88mh" podUID="b1de55b1-797e-4e61-aeee-624aa4e20ca5" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.256033    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.256259    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.76510539s: [/var/lib/containers/storage/overlay/c88b06a6ac5b65127790088be94ed578e0bc6bf9e1163adfa0d4f4c7ff4d3d3b/diff /var/log/pods/opendatahub_maas-controller-5bf8c4d6cc-v9np2_fd34cf22-51b7-4c47-969d-d8f364a35a38/manager/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.256363    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.78863346s: [/var/lib/containers/storage/overlay/4fcb3ea1ff0692f55fe2a0a6f8e5b797424cb92ae2a196e90ac95d43304710a0/diff /var/log/pods/openshift-ingress_e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp_e9ab521b-2772-44a2-9f3d-a4acc73076dc/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.256405    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.256845    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.257039    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.257077    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.259205    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.259239    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.263034    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.270291    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.271306    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.271338    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.273894    5797 ???:1] "http: TLS handshake error from 10.134.0.19:35626: EOF"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.421942    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.422011    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.422773    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.422812    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.423205    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:17.545640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.423413    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.423432    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.423772    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.423796    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.424281    5797 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.431967572s: [/var/lib/containers/storage/overlay/280182971e21c1eeb77e85df68edce4b9fd1c9064be6cea063808ee1743d7d42/diff /var/log/pods/openshift-ingress_e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl_32a58013-f4b6-44c6-8579-6a14505c8be5/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.521301    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.537108    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.537162    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.578090    5797 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-597546666c-728zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.578147    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podUID="0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.628378    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.651149    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.651191    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.669092    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.669137    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.673574    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.835136    5797 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.835188    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.862713    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.883119    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.883164    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.902105    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.902112    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.902149    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:50.902184    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.222521    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.222521    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.224215    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.225292    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.241648    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.243415    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.256398    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.257641    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.257656    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": context deadline exceeded"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.262210    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.262243    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.263118    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.263146    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.266087    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.266091    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.271101    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.282103    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.282136    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.284101    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.284140    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.284136    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.284179    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.287070    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.287105    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.287072    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.287146    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.292094    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.292130    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.292097    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.292168    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.293102    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.293126    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.294073    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.294104    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.294077    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.294126    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.294146    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.294159    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.296086    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.296121    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.297103    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.298071    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.298098    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.299070    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.299092    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.299096    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.299128    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.302078    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.302110    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.302078    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.302156    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.303085    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.303124    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.305077    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.305116    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.308078    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.308104    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.309091    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.464144    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.464197    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.465096    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.465096    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.465138    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.465167    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558781    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558803    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podUID="9e38076f-6754-46f2-b054-f0aa1976319e" containerName="nbdb" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558817    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podUID="9e38076f-6754-46f2-b054-f0aa1976319e" containerName="ovnkube-controller" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558790    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-lbqr9" podUID="bf31dccb-516a-4a97-aa86-9277ec17154d" containerName="extractor" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558780    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podUID="9e38076f-6754-46f2-b054-f0aa1976319e" containerName="sbdb" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558926    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.558960    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.579147    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.579188    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.710096    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.710145    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.830822    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.871327    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.871376    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:51.904104    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.223655    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.223653    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.225820    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.240133    5797 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-597546666c-728zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.240193    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv" podUID="0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.241896    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.244386    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.256687    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": context deadline exceeded"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.262510    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.262548    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.263945    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.263977    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.282906    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.282941    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.285034    5797 patch_prober.go:29] interesting pod/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.285063    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.285124    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.285143    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.294417    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.294449    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.297634    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:52.347880    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.293087    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.293136    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.298410    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.300089    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.300133    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.686634    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:53.735192    5797 scope.go:122] "RemoveContainer" containerID="e7f72026cbfa956687e8d4ae8eaa5ef35e8d5344d8fd73009ac60698af8b8fbc"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.020828    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.299557    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:25:54.333157    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1m49.381s"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.429540    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430676    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:26:18.058721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430744    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430764    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430810    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430842    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430859    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"]
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430913    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430934    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"]
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430949    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.430961    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431020    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431035    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431054    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431091    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431104    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431140    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431170    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431183    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431206    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9707a4e-pzjb7"]
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431228    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431271    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431300    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431332    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431372    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431411    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431450    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431484    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431508    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431531    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431554    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431577    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431610    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431642    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431670    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431700    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431740    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431778    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431819    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431853    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431884    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431925    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.431964    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.432015    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.432053    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.432090    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.437935    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.438059    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.438108    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442437    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442500    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442572    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442615    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442715    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442764    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442815    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442894    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.442952    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443044    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443107    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443152    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443237    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443297    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443560    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.443918    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.444547    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.444730    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.445491    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.445550    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.445640    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.445698    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.445781    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446142    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446216    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446296    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446461    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446539    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446586    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446846    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.446911    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447003    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447064    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447109    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447192    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447242    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447278    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447357    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447407    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447481    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.447528    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.463980    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.465470    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.470172    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.487139    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.487182    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.488683    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="console" containerStatusID={"Type":"cri-o","ID":"222ef6ac943b62a32a382d0dc5b9737e86ce530d7c00355bdbd4df2bc2a32e60"} pod="openshift-console/console-6b8dc999f8-b5bn9" containerMessage="Container console failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.500646    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"ab64906e6f1e25c98a3865b197ac88f9dd8cfd2b5277f474cabdbf6306b12b6a"} pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.500695    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" containerID="cri-o://ab64906e6f1e25c98a3865b197ac88f9dd8cfd2b5277f474cabdbf6306b12b6a" gracePeriod=10
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.505777    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="cert-manager-webhook" containerStatusID={"Type":"cri-o","ID":"1b0bcf0780bfef3ce974eedb249d18a057312fa67d12dc6c83e4b701bf7b93fa"} pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" containerMessage="Container cert-manager-webhook failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.505833    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" containerID="cri-o://1b0bcf0780bfef3ce974eedb249d18a057312fa67d12dc6c83e4b701bf7b93fa" gracePeriod=30
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.541413    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"c6d8ec71f25539eb22e2ec7b846e61bf1e781f486d8d9836cec745e2b01c95e2"} pod="opendatahub/odh-model-controller-769875db56-jr85b" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.541485    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" containerID="cri-o://c6d8ec71f25539eb22e2ec7b846e61bf1e781f486d8d9836cec745e2b01c95e2" gracePeriod=10
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.541726    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"d06c2ae4a532a9a522536bc52369e0e323b4575b80c2ba80b0adb96c41d31890"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" containerMessage="Container csi-driver failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.541843    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-driver" containerID="cri-o://d06c2ae4a532a9a522536bc52369e0e323b4575b80c2ba80b0adb96c41d31890" gracePeriod=30
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.649153    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r7v8r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.649210    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.90:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.901461    5797 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:54.902211    5797 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.005890    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"} pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.005949    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" containerID="cri-o://4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f" gracePeriod=30
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.008188    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"ec86a724fda2cd7b5c05a30fcd3f6f5edf74851e78784af3f0018ec0addc5bad"} pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.008243    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="metrics-server" containerStatusID={"Type":"cri-o","ID":"277377df435b3a2c3074d418d2ff01b1b30f5320e301cdcd76f9450a43c2bc49"} pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" containerMessage="Container metrics-server failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.008288    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" containerID="cri-o://277377df435b3a2c3074d418d2ff01b1b30f5320e301cdcd76f9450a43c2bc49" gracePeriod=170
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.008250    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" containerID="cri-o://ec86a724fda2cd7b5c05a30fcd3f6f5edf74851e78784af3f0018ec0addc5bad" gracePeriod=10
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.009042    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"ebbd0d4ad3024ff5b077a455e8f070a59da63619a52ad9308bbe35ae9c75b526"} pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.009074    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" containerID="cri-o://ebbd0d4ad3024ff5b077a455e8f070a59da63619a52ad9308bbe35ae9c75b526" gracePeriod=10
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.009794    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"6831f5a952f85c7bb575aafa033cb2b5e1083b9b70ba505e08ad913a35e216d4"} pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.009822    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" containerID="cri-o://6831f5a952f85c7bb575aafa033cb2b5e1083b9b70ba505e08ad913a35e216d4" gracePeriod=10
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.292102    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-6ch6l" podUID="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.310100    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.311092    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:55.345101    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.301154    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.317606    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": context deadline exceeded" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.317696    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.327533    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.327584    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.327640    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.327664    5797 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-186.us-east-2.compute.internal" event="NodeReady"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.343200    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.345340    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.357212    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.357631    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.360298    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.370107    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.370204    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.370139    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.373943    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.383197    5797 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.383263    5797 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:56.917505    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f5f427-d647-461a-85ca-f1ca684cac4a" path="/var/lib/kubelet/pods/22f5f427-d647-461a-85ca-f1ca684cac4a/volumes"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.256227    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.318024    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.318297    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.345637    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.370588    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.384116    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.404135    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.404205    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.411099    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.411135    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.412127    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.421099    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.481229    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.489901    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.491429    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.576771    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.693943    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:57.811184    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.014208    5797 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-zcsk7 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=error getting info from node-registrar socket: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.014290    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:25:58.036002    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.565s"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.036174    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.036249    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.036277    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.036305    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:58.036380    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.416493    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.435171    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.435301    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.435351    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.571254    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjqs2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.572778    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-6ch6l" podUID="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:25:59.572907    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:00.897627    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:00.898126    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:05.304470    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.555771    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-6ch6l" podUID="d6eb6955-2fed-4ca2-9b74-e278bb5eb79c" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.888176    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.46:50052: i/o timeout"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.918419    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal" podUID="388b6cc67d09cb3e8a1219d1e84089fb" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:02.448926    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd" podUID="5d133c42-6f8e-420e-861b-37d8e673161b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.555901    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjqs2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.410912    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.136:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.591093    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.411528    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.864079    5797 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r7v8r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.411581    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.90:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:13.919925    5797 patch_prober.go:29] interesting pod/dns-default-z88mh container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.411669    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-z88mh" podUID="b1de55b1-797e-4e61-aeee-624aa4e20ca5" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.316061    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.411723    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.412983    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.413210    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.413242    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:26:18.698034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.413368    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podUID="9e38076f-6754-46f2-b054-f0aa1976319e" containerName="ovnkube-controller" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.413930    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": dial tcp 10.134.0.21:8081: connect: connection refused (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.422464    5797 trace.go:236] Trace[784455943]: "iptables ChainExists" (04-Nov-2026 01:25:50.299) (total time: 7049ms):
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: Trace[784455943]: [7.049623443s] [7.049623443s] END
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.424334    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": EOF"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.428449    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.428799    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.430742    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.432773    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.432808    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.432872    5797 patch_prober.go:29] interesting pod/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.432903    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.435943    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.50:50052: i/o timeout"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:12.686587    5797 patch_prober.go:29] interesting pod/dns-default-z88mh container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.437100    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-z88mh" podUID="b1de55b1-797e-4e61-aeee-624aa4e20ca5" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.437339    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.458441    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.820280    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.849449    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.849503    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.850223    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-lbqr9" podUID="bf31dccb-516a-4a97-aa86-9277ec17154d" containerName="extractor" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.850963    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.851172    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.851249    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.851380    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podUID="9e38076f-6754-46f2-b054-f0aa1976319e" containerName="sbdb" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.851472    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5fq7t" podUID="9e38076f-6754-46f2-b054-f0aa1976319e" containerName="nbdb" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.872268    5797 patch_prober.go:29] interesting pod/metrics-server-5bb86d5d94-r4w4m container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: [+]log ok
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: [+]poststarthook/max-in-flight-filter ok
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: [+]poststarthook/storage-object-count-tracker-hook ok
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: [-]metric-collection-timely failed: reason withheld
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: [+]metadata-informer-sync ok
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: livez check failed
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.872321    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" podUID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.932254    5797 patch_prober.go:29] interesting pod/jobset-controller-manager-775c47d899-9l8hb container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused" start-of-body=
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:14.932303    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" podUID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": dial tcp 10.134.0.35:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.229843    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.244389    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": read tcp 10.134.0.2:47846->10.134.0.26:8081: read: connection reset by peer"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.407438    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9" podUID="270dc588-8548-488f-a219-e745a8d110db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.439431    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:15.447329    5797 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="17.411s"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.447395    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.447446    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472479    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472520    5797 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472541    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472587    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472603    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472636    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-597546666c-728zv"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.472799    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" containerID="cri-o://acd2bd6a5abecae699ce96e4d442480b7768c07a109aa495e89aec122694799b" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476104    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476142    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476168    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3504be9a-5wddh"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476199    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476216    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476242    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476270    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476297    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476322    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"]
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476354    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476384    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476410    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476426    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476452    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476488    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476514    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476549    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476572    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476612    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476647    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476679    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476710    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476757    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476779    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476823    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476878    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476922    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.476978    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477038    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477069    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477109    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477159    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477176    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477214    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477246    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477277    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477308    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477343    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477391    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477475    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.477537    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.478247    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.478331    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.478376    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.478422    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-pqxmd"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.478468    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.499253    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": dial tcp 10.134.0.23:6080: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.521500    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" containerID="cri-o://b3e2aa22ac64b672ec3f91e951b78ea849992351666acd99b570c7f81f4d10d7" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.522163    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" containerID="cri-o://46c6dee6e591a6bf6be9bd8987bf1d37e614c76e776fdbf31e611fa4c55e4e49" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.527547    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" containerID="cri-o://353c3eb8a4ef9c8a69ea8ea3c4632e0c9719274de0a950b8220e0cd021a8e550" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.529550    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy" containerID="cri-o://9174a9d3e91e722c829ca91a53bd9c3e5300363fe24dbd12001ec72623e26528" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.529692    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.543444    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"778a02dcb4ffc9bd4ea192d083404f9c732dfb9e612d972ab4bdf34781d8962c"} pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.543499    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" containerID="cri-o://778a02dcb4ffc9bd4ea192d083404f9c732dfb9e612d972ab4bdf34781d8962c" gracePeriod=10
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.545490    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.545701    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.549369    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.552946    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.553048    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qb7m9"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.553110    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.558621    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.638470    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="proxy" containerStatusID={"Type":"cri-o","ID":"e5425ac0d4e90f16ea1d24f22702cd13902fac40931733b8bdb647c0cf2965d4"} pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" containerMessage="Container proxy failed liveness probe, will be restarted"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.638543    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" containerID="cri-o://e5425ac0d4e90f16ea1d24f22702cd13902fac40931733b8bdb647c0cf2965d4" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.771595    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.974709    5797 trace.go:236] Trace[759419327]: "Calculate volume metrics of data for pod odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" (04-Nov-2026 01:25:50.298) (total time: 25676ms):
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: Trace[759419327]: [25.676654008s] [25.676654008s] END
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:15.976934    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.091724    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.091874    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.264365    5797 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.344384    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": dial tcp 10.134.0.26:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.358512    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" podUID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": dial tcp 10.134.0.24:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.358550    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": dial tcp 10.134.0.23:6080: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.360970    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": dial tcp 10.134.0.31:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.371069    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": dial tcp 10.134.0.34:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.378509    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": dial tcp 10.134.0.21:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.439780    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" podUID="13f0c474-26fd-4004-8b30-14358246c509" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.32:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.623909    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.744937    5797 generic.go:356] "Generic (PLEG): container finished" podID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerID="1b0bcf0780bfef3ce974eedb249d18a057312fa67d12dc6c83e4b701bf7b93fa" exitCode=0
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.819786    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": dial tcp 10.134.0.33:8081: connect: connection refused"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.831091    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.868904    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8a7b7a-fdf6-41d6-8406-9debf8a531a2" path="/var/lib/kubelet/pods/ea8a7b7a-fdf6-41d6-8406-9debf8a531a2/volumes"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.872243    5797 generic.go:356] "Generic (PLEG): container finished" podID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerID="6831f5a952f85c7bb575aafa033cb2b5e1083b9b70ba505e08ad913a35e216d4" exitCode=0
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.927570    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.927631    5797 generic.go:356] "Generic (PLEG): container finished" podID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerID="d06c2ae4a532a9a522536bc52369e0e323b4575b80c2ba80b0adb96c41d31890" exitCode=2
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:16.935756    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" containerID="cri-o://0c77d05a6da493ba166640cf0c8ba17f625c3fe458df8aa43850a2d8935c1fee" gracePeriod=30
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:17.058241    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.193476    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.193546    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" event={"ID":"69fdf072-b1d9-4eca-83dd-fc22311d4a32","Type":"ContainerDied","Data":"1b0bcf0780bfef3ce974eedb249d18a057312fa67d12dc6c83e4b701bf7b93fa"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.193575    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" event={"ID":"3dba5003-a608-45f0-bee9-a0597e2d39b4","Type":"ContainerDied","Data":"6831f5a952f85c7bb575aafa033cb2b5e1083b9b70ba505e08ad913a35e216d4"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.193604    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerDied","Data":"d06c2ae4a532a9a522536bc52369e0e323b4575b80c2ba80b0adb96c41d31890"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.245924    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.353773    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.552221    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.606428    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.611079    5797 scope.go:122] "RemoveContainer" containerID="2a6a4fe5e10cc7423a5f5525a1dc4f282049071026cb5b1939ed86d62573420c"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.817606    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.941247    5797 generic.go:356] "Generic (PLEG): container finished" podID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerID="c6d8ec71f25539eb22e2ec7b846e61bf1e781f486d8d9836cec745e2b01c95e2" exitCode=137
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.941320    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jr85b" event={"ID":"8931f2f8-db63-47c5-9e05-15b4a0eb9c51","Type":"ContainerDied","Data":"c6d8ec71f25539eb22e2ec7b846e61bf1e781f486d8d9836cec745e2b01c95e2"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.943080    5797 generic.go:356] "Generic (PLEG): container finished" podID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerID="ebbd0d4ad3024ff5b077a455e8f070a59da63619a52ad9308bbe35ae9c75b526" exitCode=137
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.943142    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" event={"ID":"b96933e9-892e-41b6-9b0f-9568cb44c538","Type":"ContainerDied","Data":"ebbd0d4ad3024ff5b077a455e8f070a59da63619a52ad9308bbe35ae9c75b526"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.944704    5797 generic.go:356] "Generic (PLEG): container finished" podID="bd717794-dd18-4bb7-8dfe-222545f12ff7" containerID="ec86a724fda2cd7b5c05a30fcd3f6f5edf74851e78784af3f0018ec0addc5bad" exitCode=1
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.944776    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" event={"ID":"bd717794-dd18-4bb7-8dfe-222545f12ff7","Type":"ContainerDied","Data":"ec86a724fda2cd7b5c05a30fcd3f6f5edf74851e78784af3f0018ec0addc5bad"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.956493    5797 generic.go:356] "Generic (PLEG): container finished" podID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerID="4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f" exitCode=0
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.956595    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerDied","Data":"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.956648    5797 scope.go:122] "RemoveContainer" containerID="6d50afa4d6c1ee46feab2c9a1390888b49cf4169d5e40897dbf8a7566afab311"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.958637    5797 generic.go:356] "Generic (PLEG): container finished" podID="f493fdd3-b7e8-493f-af2e-df12f9651df0" containerID="ab64906e6f1e25c98a3865b197ac88f9dd8cfd2b5277f474cabdbf6306b12b6a" exitCode=0
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.958704    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" event={"ID":"f493fdd3-b7e8-493f-af2e-df12f9651df0","Type":"ContainerDied","Data":"ab64906e6f1e25c98a3865b197ac88f9dd8cfd2b5277f474cabdbf6306b12b6a"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.960544    5797 generic.go:356] "Generic (PLEG): container finished" podID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerID="778a02dcb4ffc9bd4ea192d083404f9c732dfb9e612d972ab4bdf34781d8962c" exitCode=0
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:17.960703    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" event={"ID":"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc","Type":"ContainerDied","Data":"778a02dcb4ffc9bd4ea192d083404f9c732dfb9e612d972ab4bdf34781d8962c"}
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:18.409201    5797 scope.go:122] "RemoveContainer" containerID="e405010b2eb2a72aaf3d0daa267df5d63d2fcd5f99c9d73d76619003a642ee27"
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:18.945365    5797 trace.go:236] Trace[1052543780]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" (04-Nov-2026 01:24:35.434) (total time: 103511ms):
Nov 04 01:26:19.334642 ip-10-0-140-186 kubenswrapper[5797]: Trace[1052543780]: [1m43.511042518s] [1m43.511042518s] END
Nov 04 01:26:19.930529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:19.396391    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:26:19.930529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:19.396467    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:26:19.930529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:19.615441    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:20.208979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:20.176962    5797 scope.go:122] "RemoveContainer" containerID="1812529085bdee847df2d9e97997322d0fcbdbacfbf630aba8bb1ad9c5095eaa"
Nov 04 01:26:20.795168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:20.795134    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 04 01:26:20.795324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:20.795224    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 04 01:26:20.950865 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:20.916107    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a85dd86_0f39_49f1_ac8e_0986ee8962b8.slice/crio-46c6dee6e591a6bf6be9bd8987bf1d37e614c76e776fdbf31e611fa4c55e4e49.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da19693_73e3_4bb8_9637_c68ec381334e.slice/crio-b3e2aa22ac64b672ec3f91e951b78ea849992351666acd99b570c7f81f4d10d7.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 01:26:21.357709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:21.357666    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" podUID="69fdf072-b1d9-4eca-83dd-fc22311d4a32" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": dial tcp 10.134.0.23:6080: connect: connection refused"
Nov 04 01:26:21.358119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:21.357785    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:26:21.572818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:21.548817    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output="command timed out"
Nov 04 01:26:21.819633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:21.819541    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": dial tcp 10.134.0.33:8081: connect: connection refused"
Nov 04 01:26:23.052339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.052299    5797 generic.go:356] "Generic (PLEG): container finished" podID="0da19693-73e3-4bb8-9637-c68ec381334e" containerID="b3e2aa22ac64b672ec3f91e951b78ea849992351666acd99b570c7f81f4d10d7" exitCode=0
Nov 04 01:26:23.052797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.052378    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" event={"ID":"0da19693-73e3-4bb8-9637-c68ec381334e","Type":"ContainerDied","Data":"b3e2aa22ac64b672ec3f91e951b78ea849992351666acd99b570c7f81f4d10d7"}
Nov 04 01:26:23.054946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.054921    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" event={"ID":"b96933e9-892e-41b6-9b0f-9568cb44c538","Type":"ContainerStarted","Data":"a84e1b5e75a8d96ad6b12a710b983a642b5a897e16cd1cb5e95245b0e8236dd3"}
Nov 04 01:26:23.057004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.056971    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb" event={"ID":"bd717794-dd18-4bb7-8dfe-222545f12ff7","Type":"ContainerStarted","Data":"2fc56b8c2cc298b8b100ff15eee41a0343be0c35ca959d99f8d1acdb903cc75d"}
Nov 04 01:26:23.058459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.058437    5797 generic.go:356] "Generic (PLEG): container finished" podID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerID="46c6dee6e591a6bf6be9bd8987bf1d37e614c76e776fdbf31e611fa4c55e4e49" exitCode=0
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.058501    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" event={"ID":"0a85dd86-0f39-49f1-ac8e-0986ee8962b8","Type":"ContainerDied","Data":"46c6dee6e591a6bf6be9bd8987bf1d37e614c76e776fdbf31e611fa4c55e4e49"}
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.060024    5797 generic.go:356] "Generic (PLEG): container finished" podID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerID="acd2bd6a5abecae699ce96e4d442480b7768c07a109aa495e89aec122694799b" exitCode=0
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.060096    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" event={"ID":"79591ee2-8271-4d0a-b6f8-e4bac234b112","Type":"ContainerDied","Data":"acd2bd6a5abecae699ce96e4d442480b7768c07a109aa495e89aec122694799b"}
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.062009    5797 generic.go:356] "Generic (PLEG): container finished" podID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerID="353c3eb8a4ef9c8a69ea8ea3c4632e0c9719274de0a950b8220e0cd021a8e550" exitCode=0
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.062083    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" event={"ID":"32a58013-f4b6-44c6-8579-6a14505c8be5","Type":"ContainerDied","Data":"353c3eb8a4ef9c8a69ea8ea3c4632e0c9719274de0a950b8220e0cd021a8e550"}
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.062132    5797 scope.go:122] "RemoveContainer" containerID="83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.063649    5797 generic.go:356] "Generic (PLEG): container finished" podID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerID="9174a9d3e91e722c829ca91a53bd9c3e5300363fe24dbd12001ec72623e26528" exitCode=0
Nov 04 01:26:23.191215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.063716    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" event={"ID":"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097","Type":"ContainerDied","Data":"9174a9d3e91e722c829ca91a53bd9c3e5300363fe24dbd12001ec72623e26528"}
Nov 04 01:26:23.313216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:23.313189    5797 scope.go:122] "RemoveContainer" containerID="92688b29d5149f8d4073f90889015340f25db57ff2f31c42540358bcfac6b71b"
Nov 04 01:26:24.024792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.024580    5797 scope.go:122] "RemoveContainer" containerID="d329b2d9b88e01f9b0ea176d5051ac88adb8442e8679192b24c65c98e86292e7"
Nov 04 01:26:24.073551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.073520    5797 generic.go:356] "Generic (PLEG): container finished" podID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerID="0c77d05a6da493ba166640cf0c8ba17f625c3fe458df8aa43850a2d8935c1fee" exitCode=0
Nov 04 01:26:24.073917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.073591    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" event={"ID":"d6f64df9-087e-45ed-b182-50ce24b355ba","Type":"ContainerDied","Data":"0c77d05a6da493ba166640cf0c8ba17f625c3fe458df8aa43850a2d8935c1fee"}
Nov 04 01:26:24.076588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.076557    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz" event={"ID":"69fdf072-b1d9-4eca-83dd-fc22311d4a32","Type":"ContainerStarted","Data":"a273422a2ccdf22bfd0df7f5c10154a21afc728cdab34fde1f561f0c1f93197d"}
Nov 04 01:26:24.164056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.076599    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:26:24.164056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.077895    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:26:24.164056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.078142    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb" podUID="b96933e9-892e-41b6-9b0f-9568cb44c538" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": dial tcp 10.134.0.26:8081: connect: connection refused"
Nov 04 01:26:24.356595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.356563    5797 scope.go:122] "RemoveContainer" containerID="86151189bae7c0be2abf0d0bccb8ef3ff2bedc814f4271d03bcfef4d375637dd"
Nov 04 01:26:24.478229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.478081    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 04 01:26:24.478229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.478152    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 04 01:26:24.480377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.480275    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 04 01:26:24.480377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.480321    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 04 01:26:24.571155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.481828    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 04 01:26:24.571155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.481866    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 04 01:26:24.718076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:24.718044    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:25.111547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.088966    5797 scope.go:122] "RemoveContainer" containerID="7dc8f1616641d62ec5c5c084cfcdf0c7ce148c0b863cedf9139ea6bbc6688a57"
Nov 04 01:26:25.111547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.099285    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr" event={"ID":"0da19693-73e3-4bb8-9637-c68ec381334e","Type":"ContainerDied","Data":"7dc3de28bb1d482c5398218a974cbbc8fc943f60ae3188a4fbdb5ac737fd2db5"}
Nov 04 01:26:25.111547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.099341    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dc3de28bb1d482c5398218a974cbbc8fc943f60ae3188a4fbdb5ac737fd2db5"
Nov 04 01:26:25.111547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.105323    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd" event={"ID":"f493fdd3-b7e8-493f-af2e-df12f9651df0","Type":"ContainerStarted","Data":"0c30de2553e5343615eae771612c280f57f460e177da331645b551eaebbfe0b6"}
Nov 04 01:26:25.111547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.105404    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:26:25.206308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.205927    5797 scope.go:122] "RemoveContainer" containerID="fdebb6c6a69c931c2a95c8b5ab68b99f4620eccc207fe83f00a5c8d3d5601679"
Nov 04 01:26:25.357923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.351219    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:26:25.364613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.364577    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f" podUID="35506939-7454-4450-9d9b-687b5caaea77" containerName="postgres" probeResult="failure" output=""
Nov 04 01:26:25.435757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.435719    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.435928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.435903    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.435971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.435963    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.436050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.436038    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.436203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.436191    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.436268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.436213    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.436268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.436240    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.436348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.436304    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.436348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.436322    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.473257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.473156    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.473428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.473393    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.473534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.473491    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.474760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.474720    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:25.476197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.476172    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:26:25.476724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.476692    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.520171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.520132    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:25.520348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.520193    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:25.538046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538016    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538101    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538290    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538324    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538387    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538428    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538447    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538499    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.538584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.538572    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy\") pod \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\" (UID: \"0a85dd86-0f39-49f1-ac8e-0986ee8962b8\") "
Nov 04 01:26:25.539411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.539386    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.539525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.539414    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.539525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.539429    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da19693-73e3-4bb8-9637-c68ec381334e-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.539525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.539443    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.539525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.539457    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.611734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.611656    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.611916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.611777    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.612041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.611982    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.612312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.612118    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:25.614280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.614252    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:26:25.631487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.631460    5797 scope.go:122] "RemoveContainer" containerID="83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"
Nov 04 01:26:25.631595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.631558    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.635431 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:25.632921    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a\": container with ID starting with 83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a not found: ID does not exist" containerID="83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"
Nov 04 01:26:25.635431 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:25.632971    5797 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a\": rpc error: code = NotFound desc = could not find container \"83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a\": container with ID starting with 83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a not found: ID does not exist" containerID="83b842685ea430a609392051470bdbdf981f7e2ca2f274b8c4484cc92def109a"
Nov 04 01:26:25.635431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.633012    5797 scope.go:122] "RemoveContainer" containerID="bba19a1f37a9b93fab16783c72838adee7221aef4dc8d2d8aecfffdd7a14acaa"
Nov 04 01:26:25.641142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.640817    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641141    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641265    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641292    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641425    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641552    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641615    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641793    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.641870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.641828    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.642737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.642708    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.642737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.642734    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.642857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.642749    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.642857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.642763    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.642857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.642776    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.682629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.682566    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.684588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.684539    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.684711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.684685    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:26:25.684793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.684690    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"]
Nov 04 01:26:25.685087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.685062    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:25.687167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.687068    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.687167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.687138    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.732683 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:26:25.719379    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e762190_ffec_4b55_b757_335f8cadb45a.slice/crio-ea8bde1dc73e7a280b58fa797ae17410f03b7420818f44df9b30239b86464f7f WatchSource:0}: Error finding container ea8bde1dc73e7a280b58fa797ae17410f03b7420818f44df9b30239b86464f7f: Status 404 returned error can't find the container with id ea8bde1dc73e7a280b58fa797ae17410f03b7420818f44df9b30239b86464f7f
Nov 04 01:26:25.732683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.724731    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.741726    5797 scope.go:122] "RemoveContainer" containerID="65c420302873ad657d4dd6ed001726793ac7d15a344f7c8242de7c9bce710fd8"
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.742603    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.742947    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.742961    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.742961    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743430    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743474    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743493    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743607    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743654    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743733    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743766    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:26:25.743733    5797 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0da19693-73e3-4bb8-9637-c68ec381334e/volumes/kubernetes.io~downward-api/istio-podinfo
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743791    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743796    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743844    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743881    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\") pod \"32a58013-f4b6-44c6-8579-6a14505c8be5\" (UID: \"32a58013-f4b6-44c6-8579-6a14505c8be5\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.743958    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744002    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:26:25.744013    5797 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/32a58013-f4b6-44c6-8579-6a14505c8be5/volumes/kubernetes.io~projected/istio-token
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744031    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744058    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744068    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\") pod \"0da19693-73e3-4bb8-9637-c68ec381334e\" (UID: \"0da19693-73e3-4bb8-9637-c68ec381334e\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744125    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744156    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744191    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744230    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744297    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744348    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket\") pod \"79591ee2-8271-4d0a-b6f8-e4bac234b112\" (UID: \"79591ee2-8271-4d0a-b6f8-e4bac234b112\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:26:25.744411    5797 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0da19693-73e3-4bb8-9637-c68ec381334e/volumes/kubernetes.io~empty-dir/istio-envoy
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744422    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.744483    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy\") pod \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\" (UID: \"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097\") "
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745258    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745275    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745284    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a58013-f4b6-44c6-8579-6a14505c8be5-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745293    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745306    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745317    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745325    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da19693-73e3-4bb8-9637-c68ec381334e-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745335    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.745345    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da19693-73e3-4bb8-9637-c68ec381334e-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.751627    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "kube-api-access-w5kgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.758928    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.759022    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.759060    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "kube-api-access-kvtq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.759132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.759105    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.787088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.766017    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl" pod "32a58013-f4b6-44c6-8579-6a14505c8be5" (UID: "32a58013-f4b6-44c6-8579-6a14505c8be5"). InnerVolumeSpecName "kube-api-access-x8dcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.787088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.766033    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo" pod "0a85dd86-0f39-49f1-ac8e-0986ee8962b8" (UID: "0a85dd86-0f39-49f1-ac8e-0986ee8962b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.787088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.766070    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token" pod "0da19693-73e3-4bb8-9637-c68ec381334e" (UID: "0da19693-73e3-4bb8-9637-c68ec381334e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.787088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.771586    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:26:25.814491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.814436    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.825279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.824583    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.833284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.832570    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.844466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.844413    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:25.846799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.846762    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.846940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.846855    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.846940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.846914    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.847135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.846964    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.847135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.847046    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.847240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.847206    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.847294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.847240    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.847409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.847391    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.847505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.847477    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs\") pod \"d6f64df9-087e-45ed-b182-50ce24b355ba\" (UID: \"d6f64df9-087e-45ed-b182-50ce24b355ba\") "
Nov 04 01:26:25.848308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.848269    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.849322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849272    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849304    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849320    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849336    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849349    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtq6\" (UniqueName: \"kubernetes.io/projected/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-kube-api-access-kvtq6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849362    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849378    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a85dd86-0f39-49f1-ac8e-0986ee8962b8-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849394    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849407    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849420    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a58013-f4b6-44c6-8579-6a14505c8be5-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849434    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dcl\" (UniqueName: \"kubernetes.io/projected/32a58013-f4b6-44c6-8579-6a14505c8be5-kube-api-access-x8dcl\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849447    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5kgt\" (UniqueName: \"kubernetes.io/projected/0da19693-73e3-4bb8-9637-c68ec381334e-kube-api-access-w5kgt\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.849536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.849461    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79591ee2-8271-4d0a-b6f8-e4bac234b112-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.877520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.877447    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.918728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.889326    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.918728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.897370    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.919048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.919011    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.919135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.919116    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.919198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.919152    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "kube-api-access-8g8s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.919507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.919475    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.920655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.920619    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.921136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.921022    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.928614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.927943    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:25.928614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.928007    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.936102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.936063    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.951129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.950226    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952259    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952291    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952305    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952318    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952330    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952344    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8s4\" (UniqueName: \"kubernetes.io/projected/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-kube-api-access-8g8s4\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952358    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79591ee2-8271-4d0a-b6f8-e4bac234b112-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952371    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952383    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952396    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952408    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952421    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79591ee2-8271-4d0a-b6f8-e4bac234b112-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.952433    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.955107    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.955180    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.965906    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs" pod "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" (UID: "bc678ad4-1fa8-4e20-ab0c-f35e2d60e097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.970688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.968290    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:25.971723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.971530    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:25.971723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.971624    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:25.971723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.971698    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:25.972500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.972410    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:25.972588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.972498    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "kube-api-access-fvnc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.991934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.991837    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:25.991934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.991896    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd" pod "79591ee2-8271-4d0a-b6f8-e4bac234b112" (UID: "79591ee2-8271-4d0a-b6f8-e4bac234b112"). InnerVolumeSpecName "kube-api-access-j76pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:25.992301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:25.991979    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token" pod "d6f64df9-087e-45ed-b182-50ce24b355ba" (UID: "d6f64df9-087e-45ed-b182-50ce24b355ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:26.021607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.021584    5797 scope.go:122] "RemoveContainer" containerID="6287560de98dc57e73ae890b17ab42319fa25ffb0920ab338742aa3075943983"
Nov 04 01:26:26.056794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.056694    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.056794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.056732    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnc6\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-kube-api-access-fvnc6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.056794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.056748    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.057938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.057828    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.057938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.057860    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.057938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.057885    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f64df9-087e-45ed-b182-50ce24b355ba-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.057938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.057900    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76pd\" (UniqueName: \"kubernetes.io/projected/79591ee2-8271-4d0a-b6f8-e4bac234b112-kube-api-access-j76pd\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.057938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.057910    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.057938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.057919    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f64df9-087e-45ed-b182-50ce24b355ba-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:26.120756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.120700    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2" event={"ID":"0a85dd86-0f39-49f1-ac8e-0986ee8962b8","Type":"ContainerDied","Data":"8deb9516fd4121711f2ef9a44dab32a9929b067c263fe092ee48d2591e108ae5"}
Nov 04 01:26:26.121158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.120780    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"
Nov 04 01:26:26.121158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.120825    5797 scope.go:122] "RemoveContainer" containerID="46c6dee6e591a6bf6be9bd8987bf1d37e614c76e776fdbf31e611fa4c55e4e49"
Nov 04 01:26:26.131246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.131217    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd" event={"ID":"d6f64df9-087e-45ed-b182-50ce24b355ba","Type":"ContainerDied","Data":"c5ca5a9a396fb4fbd0ede4002bd499c6b600550cd2973739defefe9d4783acfb"}
Nov 04 01:26:26.131558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.131532    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"
Nov 04 01:26:26.139033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.139006    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" event={"ID":"1e762190-ffec-4b55-b757-335f8cadb45a","Type":"ContainerStarted","Data":"ea8bde1dc73e7a280b58fa797ae17410f03b7420818f44df9b30239b86464f7f"}
Nov 04 01:26:26.140415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.140391    5797 scope.go:122] "RemoveContainer" containerID="bc15ec25e9ad4e7ce6e7028021d06f405183426582933dfc3b3803294b6289ef"
Nov 04 01:26:26.140732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.140711    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"
Nov 04 01:26:26.140826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.140727    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst" event={"ID":"79591ee2-8271-4d0a-b6f8-e4bac234b112","Type":"ContainerDied","Data":"fdeee03385da797afcc8b89b2f0a13ace64444ee7beefbefba6988a75cb3e900"}
Nov 04 01:26:26.144534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.144241    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"
Nov 04 01:26:26.161436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.144241    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl" event={"ID":"32a58013-f4b6-44c6-8579-6a14505c8be5","Type":"ContainerDied","Data":"2ce9c9a9633a19e4635ada2132c8e731477bebcae21223e3ec052d36ea9a3fa5"}
Nov 04 01:26:26.161436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.156794    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"
Nov 04 01:26:26.161436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.156809    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6" event={"ID":"bc678ad4-1fa8-4e20-ab0c-f35e2d60e097","Type":"ContainerDied","Data":"b95a5c2de698e523d23312d3e55b13aced1d30bb36e7a63336b0a8717d109b90"}
Nov 04 01:26:26.161436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.156794    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"
Nov 04 01:26:26.174874 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.169675    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"]
Nov 04 01:26:26.176662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.176638    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nzlz2"]
Nov 04 01:26:26.194459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.194431    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"]
Nov 04 01:26:26.204138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.204102    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-zx5jd"]
Nov 04 01:26:26.207278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.207255    5797 scope.go:122] "RemoveContainer" containerID="0c77d05a6da493ba166640cf0c8ba17f625c3fe458df8aa43850a2d8935c1fee"
Nov 04 01:26:26.224359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.224325    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"]
Nov 04 01:26:26.228793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.228770    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-jgbst"]
Nov 04 01:26:26.249894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.249788    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"]
Nov 04 01:26:26.256460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.256425    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-9kxcr"]
Nov 04 01:26:26.267863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.266491    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"]
Nov 04 01:26:26.269209 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:26.269177    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:26:26.271886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.271845    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-96pcl"]
Nov 04 01:26:26.281167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.273512    5797 scope.go:122] "RemoveContainer" containerID="829a8f305ab40043cc9c84824cbed955584a47047991c81badb01ac30ecfcd3f"
Nov 04 01:26:26.289328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.289280    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"]
Nov 04 01:26:26.293305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.293283    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-wwqc6"]
Nov 04 01:26:26.341453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.341394    5797 scope.go:122] "RemoveContainer" containerID="acd2bd6a5abecae699ce96e4d442480b7768c07a109aa495e89aec122694799b"
Nov 04 01:26:26.360970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.360940    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": dial tcp 10.134.0.31:8081: connect: connection refused"
Nov 04 01:26:26.370926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.370860    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-jr85b" podUID="8931f2f8-db63-47c5-9e05-15b4a0eb9c51" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/readyz\": dial tcp 10.134.0.34:8081: connect: connection refused"
Nov 04 01:26:26.377487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.374641    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" podUID="3dba5003-a608-45f0-bee9-a0597e2d39b4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.21:8081/readyz\": dial tcp 10.134.0.21:8081: connect: connection refused"
Nov 04 01:26:26.482561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.482397    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" path="/var/lib/kubelet/pods/0a85dd86-0f39-49f1-ac8e-0986ee8962b8/volumes"
Nov 04 01:26:26.482645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.482573    5797 scope.go:122] "RemoveContainer" containerID="28f680056b5c929ba62aecb3dcfa4f471c8718406e0568742e4580467fc82b27"
Nov 04 01:26:26.483151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.483125    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" path="/var/lib/kubelet/pods/0da19693-73e3-4bb8-9637-c68ec381334e/volumes"
Nov 04 01:26:26.483803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.483782    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" path="/var/lib/kubelet/pods/32a58013-f4b6-44c6-8579-6a14505c8be5/volumes"
Nov 04 01:26:26.490622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.490594    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" path="/var/lib/kubelet/pods/79591ee2-8271-4d0a-b6f8-e4bac234b112/volumes"
Nov 04 01:26:26.491301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.491280    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" path="/var/lib/kubelet/pods/bc678ad4-1fa8-4e20-ab0c-f35e2d60e097/volumes"
Nov 04 01:26:26.500571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.500515    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" path="/var/lib/kubelet/pods/d6f64df9-087e-45ed-b182-50ce24b355ba/volumes"
Nov 04 01:26:26.599519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.599493    5797 scope.go:122] "RemoveContainer" containerID="353c3eb8a4ef9c8a69ea8ea3c4632e0c9719274de0a950b8220e0cd021a8e550"
Nov 04 01:26:26.778980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.778602    5797 scope.go:122] "RemoveContainer" containerID="9174a9d3e91e722c829ca91a53bd9c3e5300363fe24dbd12001ec72623e26528"
Nov 04 01:26:26.819880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.819844    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" podUID="6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": dial tcp 10.134.0.33:8081: connect: connection refused"
Nov 04 01:26:26.913826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:26.913807    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 04 01:26:27.171782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.171742    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" event={"ID":"1e762190-ffec-4b55-b757-335f8cadb45a","Type":"ContainerStarted","Data":"b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054"}
Nov 04 01:26:27.201031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.200955    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podStartSLOduration=150.200936417 podStartE2EDuration="2m30.200936417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:27.196806841 +0000 UTC m=+1374.838713094" watchObservedRunningTime="2026-11-04 01:26:27.200936417 +0000 UTC m=+1374.842842668"
Nov 04 01:26:27.256268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.256191    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:26:27.265522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.261127    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:27.265522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.261210    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:27.411220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.411172    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:27.411411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.411242    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:27.768347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:27.768297    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" containerID="cri-o://222ef6ac943b62a32a382d0dc5b9737e86ce530d7c00355bdbd4df2bc2a32e60" gracePeriod=7
Nov 04 01:26:28.191085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.191058    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx" event={"ID":"3dba5003-a608-45f0-bee9-a0597e2d39b4","Type":"ContainerStarted","Data":"476d57e0173da50a01402066514c10e7d0dcb8426f5bcd9e2b40f68473cab840"}
Nov 04 01:26:28.194800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.191689    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:26:28.196055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.195882    5797 generic.go:356] "Generic (PLEG): container finished" podID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerID="222ef6ac943b62a32a382d0dc5b9737e86ce530d7c00355bdbd4df2bc2a32e60" exitCode=0
Nov 04 01:26:28.196055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.195937    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8dc999f8-b5bn9" event={"ID":"f51cfd3b-b1ee-4361-aef7-d80d21dac86a","Type":"ContainerDied","Data":"222ef6ac943b62a32a382d0dc5b9737e86ce530d7c00355bdbd4df2bc2a32e60"}
Nov 04 01:26:28.202615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.202307    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jr85b" event={"ID":"8931f2f8-db63-47c5-9e05-15b4a0eb9c51","Type":"ContainerStarted","Data":"c5301fadfbcd40a363b2cea9f330d0bc0431257444f82ae4dfb59e7129cb17bd"}
Nov 04 01:26:28.203667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.203593    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:26:28.206245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.206219    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerStarted","Data":"99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a"}
Nov 04 01:26:28.206367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.206276    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:26:28.214509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.214482    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p" event={"ID":"6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc","Type":"ContainerStarted","Data":"d8e7184879482a8f5416f478d537fd224398d5a020da9adad19c63a33257ace2"}
Nov 04 01:26:28.215173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.215007    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:26:28.260087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.256439    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:28.260087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.256501    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:28.404890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.404792    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:28.608698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:28.608668    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:29.224042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.223817    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8dc999f8-b5bn9" event={"ID":"f51cfd3b-b1ee-4361-aef7-d80d21dac86a","Type":"ContainerStarted","Data":"70b013fde95c0cfc63b84ea108d3390b9c547f14934f77d973180f9452d39153"}
Nov 04 01:26:29.224042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.223984    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:29.224530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.224215    5797 patch_prober.go:29] interesting pod/console-6b8dc999f8-b5bn9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body=
Nov 04 01:26:29.224530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.224265    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b8dc999f8-b5bn9" podUID="f51cfd3b-b1ee-4361-aef7-d80d21dac86a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused"
Nov 04 01:26:29.225985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.225964    5797 generic.go:356] "Generic (PLEG): container finished" podID="af20f3c0-5693-4ff4-9587-616247f7553e" containerID="cc2b338eb804b3fd1e008b956c0d923b7f15cace16a55089688388df0ddfabea" exitCode=0
Nov 04 01:26:29.226127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.226034    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" event={"ID":"af20f3c0-5693-4ff4-9587-616247f7553e","Type":"ContainerDied","Data":"cc2b338eb804b3fd1e008b956c0d923b7f15cace16a55089688388df0ddfabea"}
Nov 04 01:26:29.226750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.226731    5797 scope.go:122] "RemoveContainer" containerID="cc2b338eb804b3fd1e008b956c0d923b7f15cace16a55089688388df0ddfabea"
Nov 04 01:26:29.228566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.228540    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:29.228652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.228609    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:29.228692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.228650    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:29.267815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.266446    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:29.267815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.266502    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:29.505662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.505556    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:29.577249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.577217    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ch6l"
Nov 04 01:26:29.777855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.777744    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:29.899891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.899858    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:26:29.918850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:29.918821    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gsw2f"
Nov 04 01:26:30.111458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.111385    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ffngz"
Nov 04 01:26:30.272016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.262767    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:30.272016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.262824    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:30.304846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.304801    5797 generic.go:356] "Generic (PLEG): container finished" podID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerID="f5a4cf00c879b563379c62f69ed751cdf4158c744c2244d4d1cdc547584215bb" exitCode=0
Nov 04 01:26:30.305095 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.304948    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" event={"ID":"b3056bef-bc61-4c70-8aa9-1a0673bf5733","Type":"ContainerDied","Data":"f5a4cf00c879b563379c62f69ed751cdf4158c744c2244d4d1cdc547584215bb"}
Nov 04 01:26:30.314016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.309120    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:26:30.314016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.309320    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerStarted","Data":"7a64586c759236d78fbb8947709c216195198ef0a8b4cc5ff3c2ac6a94928142"}
Nov 04 01:26:30.314016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.310067    5797 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-node-driver-registrar" containerStatusID={"Type":"cri-o","ID":"fecffc0fdc3e7590af5bfc66d3a3d3a3d28d0fd570eaa3df510bbe6df2503f52"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" containerMessage="Container csi-node-driver-registrar failed liveness probe, will be restarted"
Nov 04 01:26:30.314016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.311483    5797 scope.go:122] "RemoveContainer" containerID="f5a4cf00c879b563379c62f69ed751cdf4158c744c2244d4d1cdc547584215bb"
Nov 04 01:26:30.314817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.314152    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:30.314817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.314240    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:30.314817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.314291    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:30.331128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.316369    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" event={"ID":"af20f3c0-5693-4ff4-9587-616247f7553e","Type":"ContainerStarted","Data":"0aa5a3330fc92eac07c122c40c7ed89be15f2249ec218a7cf53b12f0961b64b9"}
Nov 04 01:26:30.331128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.320307    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:30.331128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.330254    5797 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 04 01:26:30.332210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.332145    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" podUID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerName="csi-node-driver-registrar" containerID="cri-o://fecffc0fdc3e7590af5bfc66d3a3d3a3d28d0fd570eaa3df510bbe6df2503f52" gracePeriod=30
Nov 04 01:26:30.338929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.338905    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:30.439931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.439789    5797 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"8143c3e5-5b37-45be-9080-f9f5dfd626e0","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"}
Nov 04 01:26:30.440168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.440147    5797 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 04 01:26:30.577841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.577784    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:30.828055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.827975    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:30.972066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.971114    5797 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 04 01:26:30.995580 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.995546    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:26:30.995943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.995912    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 04 01:26:30.996249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:30.996218    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 04 01:26:31.172857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.172823    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:31.256763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.256663    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:31.257072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.257041    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:31.326359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.326316    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" event={"ID":"b3056bef-bc61-4c70-8aa9-1a0673bf5733","Type":"ContainerStarted","Data":"9a54254ca043412ffd80eee661c74f758598b33e74c2d698c512df2dc99286bc"}
Nov 04 01:26:31.329527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.329503    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:26:31.329668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.329545    5797 generic.go:356] "Generic (PLEG): container finished" podID="5083bdb1-d4a2-4162-9b49-e3191fa61637" containerID="fecffc0fdc3e7590af5bfc66d3a3d3a3d28d0fd570eaa3df510bbe6df2503f52" exitCode=0
Nov 04 01:26:31.329668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.329573    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerDied","Data":"fecffc0fdc3e7590af5bfc66d3a3d3a3d28d0fd570eaa3df510bbe6df2503f52"}
Nov 04 01:26:31.329668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.329623    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcsk7" event={"ID":"5083bdb1-d4a2-4162-9b49-e3191fa61637","Type":"ContainerStarted","Data":"11b788c746840f688a5f1ef3d7cc12a07b1d188cf9d76a224e087024dabce265"}
Nov 04 01:26:31.441107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.441057    5797 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:26:30.971137965Z","UUID":"878c8024-1281-484e-b17a-5723e735952f","Handler":null,"Name":"","Endpoint":""}
Nov 04 01:26:31.442275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.442248    5797 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 04 01:26:31.442444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.442282    5797 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 04 01:26:31.605180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.604608    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:26:31.605180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.604747    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:31.605180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.604796    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:31.963011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:31.962549    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:32.005013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.004461    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 04 01:26:32.005013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.004529    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 04 01:26:32.262072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.259368    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:32.262072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.259427    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:32.288015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.273433    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:26:32.313016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.296539    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:32.536061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.535963    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:32.592015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.591386    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:32.592015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.591449    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:32.733391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:32.733356    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:33.007667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.007593    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 04 01:26:33.007667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.007658    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 04 01:26:33.072015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.070876    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:33.280080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.279802    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:33.280080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.279869    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:33.283025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.281207    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:33.519013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.518880    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:33.592069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.591394    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:33.592069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.591458    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:33.839947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:33.839910    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:34.007678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.007641    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 04 01:26:34.007968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.007940    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 04 01:26:34.078012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.073511    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:34.079244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.079217    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:26:34.124070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.124038    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-smsgb"
Nov 04 01:26:34.268343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.268247    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 04 01:26:34.268684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.268647    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 04 01:26:34.600127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.591413    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:34.600127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.591481    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:34.995548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.995503    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 04 01:26:34.995860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:34.995826    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 04 01:26:35.136067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.134358    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-775c47d899-9l8hb"
Nov 04 01:26:35.310071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.306680    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:35.322554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.322519    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:26:35.325664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.325568    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:26:35.615076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.591139    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:35.656205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.656156    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:35.775583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:35.772858    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:36.024853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.024758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:36.182015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.181898    5797 patch_prober.go:29] interesting pod/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 04 01:26:36.182015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.181958    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 04 01:26:36.202025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.199980    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7l6kd"
Nov 04 01:26:36.506122 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.506077    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:36.523871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.514112    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:36.537011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.514148    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 04 01:26:36.611579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.611543    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:36.611862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:36.611823    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:37.144828 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:37.144783    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:26:37.145497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:37.145475    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:26:37.159330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:37.159301    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:26:37.591481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:37.591388    5797 patch_prober.go:29] interesting pod/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 04 01:26:37.591769 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:37.591736    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 04 01:26:38.596026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:38.595960    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:26:38.596450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:38.596424    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:26:38.597209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:38.597184    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:26:39.029304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:39.029273    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"]
Nov 04 01:26:39.034227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:39.034180    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2aaa5d-z6nqm"]
Nov 04 01:26:39.228491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:39.228464    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:26:39.229325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:39.229300    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-jr85b"
Nov 04 01:26:40.318773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:40.318742    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff7fd4ff-hxltx"
Nov 04 01:26:40.427978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:40.427940    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:40.506409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:40.506372    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc39046-127e-40c2-aed6-08fbcaafe393" path="/var/lib/kubelet/pods/5bc39046-127e-40c2-aed6-08fbcaafe393/volumes"
Nov 04 01:26:40.633930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:40.633895    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:41.179335 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:41.179302    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b8dc999f8-b5bn9"
Nov 04 01:26:41.558273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:41.558141    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:41.753720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:41.753685    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:41.956483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:41.956448    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:42.622973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:42.622932    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:43.789186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:43.789143    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:43.789584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:43.789231    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:44.893623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.893588    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:44.949883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.949830    5797 generic.go:356] "Generic (PLEG): container finished" podID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerID="9c02dd011ff8f37cdc4a2431185502d486e3d8d2b0cc5c4c685bd627b3bf06f3" exitCode=0
Nov 04 01:26:44.949883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.949876    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" event={"ID":"33f3f13f-1942-4dca-b993-0f48deaa503e","Type":"ContainerDied","Data":"9c02dd011ff8f37cdc4a2431185502d486e3d8d2b0cc5c4c685bd627b3bf06f3"}
Nov 04 01:26:44.950506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.950475    5797 scope.go:122] "RemoveContainer" containerID="9c02dd011ff8f37cdc4a2431185502d486e3d8d2b0cc5c4c685bd627b3bf06f3"
Nov 04 01:26:44.953201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.953147    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:44.953521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.953489    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:44.953627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:44.953555    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:26:45.109594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:45.109554    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:45.967008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:45.966894    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" event={"ID":"33f3f13f-1942-4dca-b993-0f48deaa503e","Type":"ContainerStarted","Data":"f92686dab3a0e30de9b6d71aa0c2bfd9053874fc8f5824c04075667cb1541fba"}
Nov 04 01:26:45.970961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:45.970930    5797 generic.go:356] "Generic (PLEG): container finished" podID="13f0c474-26fd-4004-8b30-14358246c509" containerID="e5425ac0d4e90f16ea1d24f22702cd13902fac40931733b8bdb647c0cf2965d4" exitCode=137
Nov 04 01:26:45.971137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:45.971028    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" event={"ID":"13f0c474-26fd-4004-8b30-14358246c509","Type":"ContainerDied","Data":"e5425ac0d4e90f16ea1d24f22702cd13902fac40931733b8bdb647c0cf2965d4"}
Nov 04 01:26:45.971137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:45.971071    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw" event={"ID":"13f0c474-26fd-4004-8b30-14358246c509","Type":"ContainerStarted","Data":"ada892dae8e9e50855498a73ddec5d8ffc7cdcfab58d745593a3de97fb6c0328"}
Nov 04 01:26:45.971341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:45.971328    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:26:46.009925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.009892    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"]
Nov 04 01:26:46.010610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010594    5797 cpu_manager.go:414] "containerMap: removing container" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010612    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010625    5797 cpu_manager.go:414] "containerMap: removing container" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010630    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010638    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010643    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010659    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy"
Nov 04 01:26:46.010663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010665    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010674    5797 cpu_manager.go:414] "containerMap: removing container" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010679    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010692    5797 cpu_manager.go:414] "containerMap: removing container" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010696    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010713    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010718    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010829    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010836    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f64df9-087e-45ed-b182-50ce24b355ba" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010844    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da19693-73e3-4bb8-9637-c68ec381334e" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010851    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a85dd86-0f39-49f1-ac8e-0986ee8962b8" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010857    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="79591ee2-8271-4d0a-b6f8-e4bac234b112" containerName="istio-proxy"
Nov 04 01:26:46.010890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.010866    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc678ad4-1fa8-4e20-ab0c-f35e2d60e097" containerName="istio-proxy"
Nov 04 01:26:46.014612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.014593    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.021435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.021388    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:26:46.021852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.021813    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:26:46.043191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.043161    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"]
Nov 04 01:26:46.155165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.150069    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.155165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.150276    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn7b\" (UniqueName: \"kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.155165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.150452    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.277169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.277068    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn7b\" (UniqueName: \"kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.277931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.277904    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.283075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.279108    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.284046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.278862    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.290637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.290551    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.291501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.291357    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn7b\" (UniqueName: \"kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b\") pod \"maas-api-revoke-sub-2d38cfb5-29m57\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.339104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.339072    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:46.645655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.645613    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:46.881014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.880319    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"]
Nov 04 01:26:46.887614 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:26:46.887573    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb409099_6c52_489b_957e_f930401ebc21.slice/crio-75a14fae19a59306308045b8d3389ddb59977d66337cb39827958101a0302ef9 WatchSource:0}: Error finding container 75a14fae19a59306308045b8d3389ddb59977d66337cb39827958101a0302ef9: Status 404 returned error can't find the container with id 75a14fae19a59306308045b8d3389ddb59977d66337cb39827958101a0302ef9
Nov 04 01:26:46.907795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.907734    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:26:46.908230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.908154    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:46.908316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.908253    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:46.998084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:46.996130    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57" event={"ID":"bb409099-6c52-489b-957e-f930401ebc21","Type":"ContainerStarted","Data":"75a14fae19a59306308045b8d3389ddb59977d66337cb39827958101a0302ef9"}
Nov 04 01:26:47.908316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:47.908277    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:47.908502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:47.908343    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:47.983757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:47.983723    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:48.045979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.045938    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57" event={"ID":"bb409099-6c52-489b-957e-f930401ebc21","Type":"ContainerStarted","Data":"d595ed7a8b5ac2b57f689e9b87546d08eb7beb1dcbcc134f0f1ce2f772d9f091"}
Nov 04 01:26:48.088062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.087955    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57" podStartSLOduration=3.087935631 podStartE2EDuration="3.087935631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:48.082095888 +0000 UTC m=+1395.724002158" watchObservedRunningTime="2026-11-04 01:26:48.087935631 +0000 UTC m=+1395.729841883"
Nov 04 01:26:48.549072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.543429    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"]
Nov 04 01:26:48.549072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.544813    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a58013-f4b6-44c6-8579-6a14505c8be5" containerName="istio-proxy"
Nov 04 01:26:48.578039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.566037    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.622860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.622772    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"]
Nov 04 01:26:48.724099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.724055    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.724300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.724261    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.724382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.724367    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4bn\" (UniqueName: \"kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.828452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.828408    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.828631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.828520    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4bn\" (UniqueName: \"kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.828904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.828857    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.829915 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.829891    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.831748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.831726    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.880756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.879752    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4bn\" (UniqueName: \"kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn\") pod \"maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:48.916013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.908517    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:48.916013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.908923    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:48.916013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.908969    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:48.941648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:48.941615    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:49.103197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.101161    5797 generic.go:356] "Generic (PLEG): container finished" podID="bb409099-6c52-489b-957e-f930401ebc21" containerID="d595ed7a8b5ac2b57f689e9b87546d08eb7beb1dcbcc134f0f1ce2f772d9f091" exitCode=0
Nov 04 01:26:49.103197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.101235    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57" event={"ID":"bb409099-6c52-489b-957e-f930401ebc21","Type":"ContainerDied","Data":"d595ed7a8b5ac2b57f689e9b87546d08eb7beb1dcbcc134f0f1ce2f772d9f091"}
Nov 04 01:26:49.193239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.192954    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"]
Nov 04 01:26:49.193463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.193434    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" containerID="cri-o://d5bcb61c5b5a47b85c4de69d1787f30f8cbc7c48ba4cf467bd9b2c3ed1897d94" gracePeriod=30
Nov 04 01:26:49.197346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.197317    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"]
Nov 04 01:26:49.197588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.197544    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" containerID="cri-o://326ec39ad4f4f95216d4b4903203e268f665c58fc8f405d46d7480cfc8ded9c5" gracePeriod=30
Nov 04 01:26:49.309033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.307614    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:49.633440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.633404    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:49.667775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.654765    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"]
Nov 04 01:26:49.832926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.832897    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:49.908700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.908617    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:49.908700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:49.908678    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:50.077018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.072514    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:50.112856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.112818    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td" event={"ID":"a7a7775b-1d7e-4c72-9960-7657a192aa1f","Type":"ContainerStarted","Data":"75afbaded4e9274ec54db082f11b801989077e9e44bd5a4d29aadd37c9ce794b"}
Nov 04 01:26:50.310807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.310705    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:50.400675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.400642    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:50.503083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.501128    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca\") pod \"bb409099-6c52-489b-957e-f930401ebc21\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") "
Nov 04 01:26:50.503083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.502074    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca\") pod \"bb409099-6c52-489b-957e-f930401ebc21\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") "
Nov 04 01:26:50.503083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.502444    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b\" (UniqueName: \"kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b\") pod \"bb409099-6c52-489b-957e-f930401ebc21\" (UID: \"bb409099-6c52-489b-957e-f930401ebc21\") "
Nov 04 01:26:50.503372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.503346    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca" pod "bb409099-6c52-489b-957e-f930401ebc21" (UID: "bb409099-6c52-489b-957e-f930401ebc21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:50.513871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.513762    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca" pod "bb409099-6c52-489b-957e-f930401ebc21" (UID: "bb409099-6c52-489b-957e-f930401ebc21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:26:50.515612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.515045    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b" pod "bb409099-6c52-489b-957e-f930401ebc21" (UID: "bb409099-6c52-489b-957e-f930401ebc21"). InnerVolumeSpecName "kube-api-access-tjn7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:50.605504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.605469    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjn7b\" (UniqueName: \"kubernetes.io/projected/bb409099-6c52-489b-957e-f930401ebc21-kube-api-access-tjn7b\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:50.605504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.605500    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb409099-6c52-489b-957e-f930401ebc21-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:50.605504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.605510    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb409099-6c52-489b-957e-f930401ebc21-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:50.930248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.865166    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:50.930248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.909551    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:50.930248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:50.909608    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:51.256013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.248222    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td" event={"ID":"a7a7775b-1d7e-4c72-9960-7657a192aa1f","Type":"ContainerStarted","Data":"02732a2ed8a59cc0f1358ab9a58260748470eb9e0470c165cf090b3b3986064f"}
Nov 04 01:26:51.260237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.260199    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57" event={"ID":"bb409099-6c52-489b-957e-f930401ebc21","Type":"ContainerDied","Data":"75a14fae19a59306308045b8d3389ddb59977d66337cb39827958101a0302ef9"}
Nov 04 01:26:51.260513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.260483    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a14fae19a59306308045b8d3389ddb59977d66337cb39827958101a0302ef9"
Nov 04 01:26:51.260746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.260726    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"
Nov 04 01:26:51.297235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.297165    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td" podStartSLOduration=3.297146433 podStartE2EDuration="3.297146433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:51.292532228 +0000 UTC m=+1398.934438478" watchObservedRunningTime="2026-11-04 01:26:51.297146433 +0000 UTC m=+1398.939052688"
Nov 04 01:26:51.579214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.579125    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:51.820011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.815004    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:51.931065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.919564    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:51.931065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:51.919638    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:52.268076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.265859    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:52.306139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.304476    5797 generic.go:356] "Generic (PLEG): container finished" podID="a7a7775b-1d7e-4c72-9960-7657a192aa1f" containerID="02732a2ed8a59cc0f1358ab9a58260748470eb9e0470c165cf090b3b3986064f" exitCode=0
Nov 04 01:26:52.306139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.304783    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td" event={"ID":"a7a7775b-1d7e-4c72-9960-7657a192aa1f","Type":"ContainerDied","Data":"02732a2ed8a59cc0f1358ab9a58260748470eb9e0470c165cf090b3b3986064f"}
Nov 04 01:26:52.500241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.500046    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:52.753015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.752932    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:52.915014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.910464    5797 patch_prober.go:29] interesting pod/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 04 01:26:52.915014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:52.910593    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 04 01:26:53.556412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.556380    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:53.661615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.661577    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca\") pod \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") "
Nov 04 01:26:53.662378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.662351    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca\") pod \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") "
Nov 04 01:26:53.662927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.662906    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn\" (UniqueName: \"kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn\") pod \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\" (UID: \"a7a7775b-1d7e-4c72-9960-7657a192aa1f\") "
Nov 04 01:26:53.663635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.662819    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca" pod "a7a7775b-1d7e-4c72-9960-7657a192aa1f" (UID: "a7a7775b-1d7e-4c72-9960-7657a192aa1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:53.669009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.664749    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:53.723525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.722243    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca" pod "a7a7775b-1d7e-4c72-9960-7657a192aa1f" (UID: "a7a7775b-1d7e-4c72-9960-7657a192aa1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:26:53.723525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.722444    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn" pod "a7a7775b-1d7e-4c72-9960-7657a192aa1f" (UID: "a7a7775b-1d7e-4c72-9960-7657a192aa1f"). InnerVolumeSpecName "kube-api-access-2v4bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:53.734574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.734540    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:53.772471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.772440    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a7775b-1d7e-4c72-9960-7657a192aa1f-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:53.772471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.772469    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4bn\" (UniqueName: \"kubernetes.io/projected/a7a7775b-1d7e-4c72-9960-7657a192aa1f-kube-api-access-2v4bn\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:53.912543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.912467    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:26:53.912922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.912892    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:26:53.913757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:53.913735    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:26:54.326960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.326929    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"
Nov 04 01:26:54.326960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.326942    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td" event={"ID":"a7a7775b-1d7e-4c72-9960-7657a192aa1f","Type":"ContainerDied","Data":"75afbaded4e9274ec54db082f11b801989077e9e44bd5a4d29aadd37c9ce794b"}
Nov 04 01:26:54.327245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.327013    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75afbaded4e9274ec54db082f11b801989077e9e44bd5a4d29aadd37c9ce794b"
Nov 04 01:26:54.333578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.333258    5797 generic.go:356] "Generic (PLEG): container finished" podID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerID="326ec39ad4f4f95216d4b4903203e268f665c58fc8f405d46d7480cfc8ded9c5" exitCode=0
Nov 04 01:26:54.333578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.333514    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" event={"ID":"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5","Type":"ContainerDied","Data":"326ec39ad4f4f95216d4b4903203e268f665c58fc8f405d46d7480cfc8ded9c5"}
Nov 04 01:26:54.352503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.352450    5797 generic.go:356] "Generic (PLEG): container finished" podID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerID="d5bcb61c5b5a47b85c4de69d1787f30f8cbc7c48ba4cf467bd9b2c3ed1897d94" exitCode=0
Nov 04 01:26:54.372240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.361478    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" event={"ID":"59912ca4-fdd9-4f5b-9713-220f3136e515","Type":"ContainerDied","Data":"d5bcb61c5b5a47b85c4de69d1787f30f8cbc7c48ba4cf467bd9b2c3ed1897d94"}
Nov 04 01:26:54.372240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.361573    5797 scope.go:122] "RemoveContainer" containerID="8d397e7ba01a8c7e6249629a6961e160a47952c7035421e5823cb0720427a086"
Nov 04 01:26:54.442052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.440144    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:54.466513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.466173    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 04 01:26:54.466513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.466244    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 04 01:26:54.630696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.630666    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:26:54.704105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.704064    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:26:54.730030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.729862    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.731754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.731376    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.731754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.731441    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.731754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.731520    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.731754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.731617    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.731754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.731647    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.731754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.731676    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.732620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.732592    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:54.732792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.732707    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.732942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.732920    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.733557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.733524    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.733756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.733733    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.733842 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.733770    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.733901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.733874    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.733956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.733921    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.733956 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.733950    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.734696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.734016    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.734696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.734050    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.734696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.734091    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.734696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.734283    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.735094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.735068    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:26:54.736293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736211    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.736479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736440    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.736650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736625    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.736727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736714    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.736850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736832    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy\") pod \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\" (UID: \"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5\") "
Nov 04 01:26:54.736960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736944    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.737048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.736956    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo\") pod \"59912ca4-fdd9-4f5b-9713-220f3136e515\" (UID: \"59912ca4-fdd9-4f5b-9713-220f3136e515\") "
Nov 04 01:26:54.737731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.737695    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:54.738108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738064    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738111    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738127    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738140    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738171    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59912ca4-fdd9-4f5b-9713-220f3136e515-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738187    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738201    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738215    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738246    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738261    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.738590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.738274    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.740668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.740613    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "kube-api-access-d6tl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:54.740668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.740648    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:54.741256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.741224    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:54.741677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.741646    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.741778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.741745    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:26:54.742799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.742771    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo" pod "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" (UID: "d4586618-b0f4-4315-ad8b-f4fdeddbb5f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:26:54.743077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.743053    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s" pod "59912ca4-fdd9-4f5b-9713-220f3136e515" (UID: "59912ca4-fdd9-4f5b-9713-220f3136e515"). InnerVolumeSpecName "kube-api-access-st82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:26:54.840590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840549    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.840590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840593    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.840772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840612    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59912ca4-fdd9-4f5b-9713-220f3136e515-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.840772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840626    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.840772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840641    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tl2\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-kube-api-access-d6tl2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.840772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840655    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:54.840772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:54.840670    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st82s\" (UniqueName: \"kubernetes.io/projected/59912ca4-fdd9-4f5b-9713-220f3136e515-kube-api-access-st82s\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:26:55.085653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.085567    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:55.364166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.364142    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"
Nov 04 01:26:55.364344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.364185    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" event={"ID":"59912ca4-fdd9-4f5b-9713-220f3136e515","Type":"ContainerDied","Data":"7252c2eff7875708b7102093fc77c9322f4208331ea962e382c21dc5e474d6b7"}
Nov 04 01:26:55.364344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.364244    5797 scope.go:122] "RemoveContainer" containerID="d5bcb61c5b5a47b85c4de69d1787f30f8cbc7c48ba4cf467bd9b2c3ed1897d94"
Nov 04 01:26:55.365994 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.365971    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"
Nov 04 01:26:55.366143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.366035    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf" event={"ID":"d4586618-b0f4-4315-ad8b-f4fdeddbb5f5","Type":"ContainerDied","Data":"39535a322bb806810e56896bd9c450dbd6cc5d6129f38bca06bcc1ac711385e4"}
Nov 04 01:26:55.381078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.381051    5797 scope.go:122] "RemoveContainer" containerID="326ec39ad4f4f95216d4b4903203e268f665c58fc8f405d46d7480cfc8ded9c5"
Nov 04 01:26:55.401129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.401096    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"]
Nov 04 01:26:55.405377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.405345    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6"]
Nov 04 01:26:55.422630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.422602    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"]
Nov 04 01:26:55.427612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.427587    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-n8jpf"]
Nov 04 01:26:55.502867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.502834    5797 patch_prober.go:29] interesting pod/e2e-mt-e7742169-openshift-default-6986f46949-dgng6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:26:55.503074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.502892    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e7742169-openshift-default-6986f46949-dgng6" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:26:55.800586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:55.800459    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:56.125084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.125053    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:56.260601 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:26:56.260544    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:26:56.323604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.323570    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:56.482647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.482563    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" path="/var/lib/kubelet/pods/59912ca4-fdd9-4f5b-9713-220f3136e515/volumes"
Nov 04 01:26:56.483350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.483333    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" path="/var/lib/kubelet/pods/d4586618-b0f4-4315-ad8b-f4fdeddbb5f5/volumes"
Nov 04 01:26:56.524199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.524167    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:56.743227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.741867    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:56.942776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:56.942749    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:57.039800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.039720    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"]
Nov 04 01:26:57.039978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.039836    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9s6bw"
Nov 04 01:26:57.040953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.040923    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy" containerID="cri-o://0aa5a3330fc92eac07c122c40c7ed89be15f2249ec218a7cf53b12f0961b64b9" gracePeriod=30
Nov 04 01:26:57.045398 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.045346    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"]
Nov 04 01:26:57.045694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.045644    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy" containerID="cri-o://b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054" gracePeriod=30
Nov 04 01:26:57.223063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.221873    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:57.931270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:57.931232    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:58.155467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:58.155436    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:58.942013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:58.925268    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:26:59.682126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:26:59.682092    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:00.322374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:00.322343    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mvf9p"
Nov 04 01:27:00.540067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:00.538051    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:01.101000 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:01.100951    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:01.521012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:01.516324    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:02.194151 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:02.194106    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e762190_ffec_4b55_b757_335f8cadb45a.slice/crio-conmon-b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 01:27:02.195082 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:02.195044    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e762190_ffec_4b55_b757_335f8cadb45a.slice/crio-conmon-b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf20f3c0_5693_4ff4_9587_616247f7553e.slice/crio-conmon-0aa5a3330fc92eac07c122c40c7ed89be15f2249ec218a7cf53b12f0961b64b9.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 01:27:02.320723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.320615    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"]
Nov 04 01:27:02.321424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321400    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a7a7775b-1d7e-4c72-9960-7657a192aa1f" containerName="revoke-keys"
Nov 04 01:27:02.321424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321421    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a7775b-1d7e-4c72-9960-7657a192aa1f" containerName="revoke-keys"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321439    5797 cpu_manager.go:414] "containerMap: removing container" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321444    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321462    5797 cpu_manager.go:414] "containerMap: removing container" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321467    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321476    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bb409099-6c52-489b-957e-f930401ebc21" containerName="revoke-keys"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321480    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb409099-6c52-489b-957e-f930401ebc21" containerName="revoke-keys"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321612    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a7775b-1d7e-4c72-9960-7657a192aa1f" containerName="revoke-keys"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321620    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321629    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4586618-b0f4-4315-ad8b-f4fdeddbb5f5" containerName="istio-proxy"
Nov 04 01:27:02.321653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.321634    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb409099-6c52-489b-957e-f930401ebc21" containerName="revoke-keys"
Nov 04 01:27:02.325809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.325784    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.333087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.332534    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:27:02.333087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.332598    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:27:02.339698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.339627    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"]
Nov 04 01:27:02.413003 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.412450    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.413003 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.412668    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9hl\" (UniqueName: \"kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.413003 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.412851    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.457230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.454404    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:27:02.537070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.521865    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.537070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.522053    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.537070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.525862    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:02.552123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.552073    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.552934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.552906    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.553311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.553294    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.553484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.553468    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.559021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.553628    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.559294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.559275    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.559606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.559587    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket\") pod \"1e762190-ffec-4b55-b757-335f8cadb45a\" (UID: \"1e762190-ffec-4b55-b757-335f8cadb45a\") "
Nov 04 01:27:02.573291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.573245    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.615427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.578099    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.615594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.612301    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.615676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.615157    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:02.615758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.594662    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.615984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.615965    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.616374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.616354    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9hl\" (UniqueName: \"kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.616617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.616602    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.616714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.616702    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.616807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.616795    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e762190-ffec-4b55-b757-335f8cadb45a-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.616890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.616879    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.617262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.617233    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.631007 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.617976    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.631007 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.624560    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.631007 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.630091    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "kube-api-access-hndhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:02.631953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.631908    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9hl\" (UniqueName: \"kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl\") pod \"maas-api-revoke-sub-fb279d1d-hw57h\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.632934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.632911    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:02.633114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.632951    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.633779 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.633751    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token" pod "1e762190-ffec-4b55-b757-335f8cadb45a" (UID: "1e762190-ffec-4b55-b757-335f8cadb45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:02.644416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.644387    5797 generic.go:356] "Generic (PLEG): container finished" podID="1e762190-ffec-4b55-b757-335f8cadb45a" containerID="b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054" exitCode=0
Nov 04 01:27:02.644616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.644601    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" event={"ID":"1e762190-ffec-4b55-b757-335f8cadb45a","Type":"ContainerDied","Data":"b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054"}
Nov 04 01:27:02.644672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.644642    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5" event={"ID":"1e762190-ffec-4b55-b757-335f8cadb45a","Type":"ContainerDied","Data":"ea8bde1dc73e7a280b58fa797ae17410f03b7420818f44df9b30239b86464f7f"}
Nov 04 01:27:02.644672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.644664    5797 scope.go:122] "RemoveContainer" containerID="b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054"
Nov 04 01:27:02.656019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.644770    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"
Nov 04 01:27:02.678911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.678871    5797 generic.go:356] "Generic (PLEG): container finished" podID="af20f3c0-5693-4ff4-9587-616247f7553e" containerID="0aa5a3330fc92eac07c122c40c7ed89be15f2249ec218a7cf53b12f0961b64b9" exitCode=0
Nov 04 01:27:02.679124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.679029    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" event={"ID":"af20f3c0-5693-4ff4-9587-616247f7553e","Type":"ContainerDied","Data":"0aa5a3330fc92eac07c122c40c7ed89be15f2249ec218a7cf53b12f0961b64b9"}
Nov 04 01:27:02.681379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.681307    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:27:02.681471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.681383    5797 scope.go:122] "RemoveContainer" containerID="b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054"
Nov 04 01:27:02.681637 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:02.681613    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054\": container with ID starting with b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054 not found: ID does not exist" containerID="b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054"
Nov 04 01:27:02.681709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.681647    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054"} err="failed to get container status \"b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054\": rpc error: code = NotFound desc = could not find container \"b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054\": container with ID starting with b022b3fd6fe70c4c43be5601c6716cd183321012f0338d3cb1a2622613a49054 not found: ID does not exist"
Nov 04 01:27:02.681709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.681670    5797 scope.go:122] "RemoveContainer" containerID="cc2b338eb804b3fd1e008b956c0d923b7f15cace16a55089688388df0ddfabea"
Nov 04 01:27:02.718081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.718052    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndhr\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-kube-api-access-hndhr\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.718081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.718081    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e762190-ffec-4b55-b757-335f8cadb45a-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.718314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.718096    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.718314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.718108    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e762190-ffec-4b55-b757-335f8cadb45a-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.718314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.718121    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e762190-ffec-4b55-b757-335f8cadb45a-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.723152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.723126    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:02.728605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.727686    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"]
Nov 04 01:27:02.735280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.735248    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-thlr5"]
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819352    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819388    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819476    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819511    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819550    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819670    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819695    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819741    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.820578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.819782    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj\") pod \"af20f3c0-5693-4ff4-9587-616247f7553e\" (UID: \"af20f3c0-5693-4ff4-9587-616247f7553e\") "
Nov 04 01:27:02.822581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.822512    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.822691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.822622    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.822893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.822857    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.823041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.822965    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.823317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.823290    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:02.823876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.823851    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "kube-api-access-lcspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:02.828719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.827925    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:02.828719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.828653    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:02.829067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.829042    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo" pod "af20f3c0-5693-4ff4-9587-616247f7553e" (UID: "af20f3c0-5693-4ff4-9587-616247f7553e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921567    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921598    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20f3c0-5693-4ff4-9587-616247f7553e-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921607    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921616    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20f3c0-5693-4ff4-9587-616247f7553e-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921624    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921632    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921640    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.921645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921648    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20f3c0-5693-4ff4-9587-616247f7553e-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:02.922108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:02.921658    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcspj\" (UniqueName: \"kubernetes.io/projected/af20f3c0-5693-4ff4-9587-616247f7553e-kube-api-access-lcspj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:03.036230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.036198    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"]
Nov 04 01:27:03.037735 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:27:03.037710    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb694a80_da92_4cab_85b5_ed1919c17da6.slice/crio-773ca2a7828738a7c423d7b08fee71fc85036bb9150c1eb98068cdf891925919 WatchSource:0}: Error finding container 773ca2a7828738a7c423d7b08fee71fc85036bb9150c1eb98068cdf891925919: Status 404 returned error can't find the container with id 773ca2a7828738a7c423d7b08fee71fc85036bb9150c1eb98068cdf891925919
Nov 04 01:27:03.415020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.414084    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:03.727052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.722969    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:03.787861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.787834    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h" event={"ID":"db694a80-da92-4cab-85b5-ed1919c17da6","Type":"ContainerStarted","Data":"773ca2a7828738a7c423d7b08fee71fc85036bb9150c1eb98068cdf891925919"}
Nov 04 01:27:03.789976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.789945    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"
Nov 04 01:27:03.791189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.789982    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj" event={"ID":"af20f3c0-5693-4ff4-9587-616247f7553e","Type":"ContainerDied","Data":"5e1c67de66a5f397808634d578e46fa3313486407d196e9b8eacba815b1f2fa6"}
Nov 04 01:27:03.791189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.790943    5797 scope.go:122] "RemoveContainer" containerID="0aa5a3330fc92eac07c122c40c7ed89be15f2249ec218a7cf53b12f0961b64b9"
Nov 04 01:27:03.846404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.846371    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"]
Nov 04 01:27:03.889089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:03.889054    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6e609c-openshift-default-7dbbb69bf9-47wwj"]
Nov 04 01:27:04.002023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.001901    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:04.182794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.182742    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"]
Nov 04 01:27:04.192157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.192124    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"]
Nov 04 01:27:04.207627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.207594    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:04.404088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.404031    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:04.481233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.481201    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" path="/var/lib/kubelet/pods/1e762190-ffec-4b55-b757-335f8cadb45a/volumes"
Nov 04 01:27:04.481797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.481781    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" path="/var/lib/kubelet/pods/af20f3c0-5693-4ff4-9587-616247f7553e/volumes"
Nov 04 01:27:04.664256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.664161    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:04.808094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.808060    5797 generic.go:356] "Generic (PLEG): container finished" podID="db694a80-da92-4cab-85b5-ed1919c17da6" containerID="e55aa749411c66a5a0591a9b9c169b85d38be01a8823dc26957ce7b5f4802711" exitCode=0
Nov 04 01:27:04.808468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.808442    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h" event={"ID":"db694a80-da92-4cab-85b5-ed1919c17da6","Type":"ContainerDied","Data":"e55aa749411c66a5a0591a9b9c169b85d38be01a8823dc26957ce7b5f4802711"}
Nov 04 01:27:04.884961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:04.884868    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:05.397012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:05.392369    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:05.591103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:05.591069    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:06.162054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.161954    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:06.261583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.261486    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca\") pod \"db694a80-da92-4cab-85b5-ed1919c17da6\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") "
Nov 04 01:27:06.261583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.261545    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl\" (UniqueName: \"kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl\") pod \"db694a80-da92-4cab-85b5-ed1919c17da6\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") "
Nov 04 01:27:06.265433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.261762    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca\") pod \"db694a80-da92-4cab-85b5-ed1919c17da6\" (UID: \"db694a80-da92-4cab-85b5-ed1919c17da6\") "
Nov 04 01:27:06.265433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.261963    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca" pod "db694a80-da92-4cab-85b5-ed1919c17da6" (UID: "db694a80-da92-4cab-85b5-ed1919c17da6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:06.265433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.262623    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:06.298014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.279123    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca" pod "db694a80-da92-4cab-85b5-ed1919c17da6" (UID: "db694a80-da92-4cab-85b5-ed1919c17da6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:06.343263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.339601    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl" pod "db694a80-da92-4cab-85b5-ed1919c17da6" (UID: "db694a80-da92-4cab-85b5-ed1919c17da6"). InnerVolumeSpecName "kube-api-access-rp9hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:06.365014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.364343    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9hl\" (UniqueName: \"kubernetes.io/projected/db694a80-da92-4cab-85b5-ed1919c17da6-kube-api-access-rp9hl\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:06.365014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.364388    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db694a80-da92-4cab-85b5-ed1919c17da6-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:06.503154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.503113    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:06.710674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.710636    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:06.926800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.926687    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"
Nov 04 01:27:06.926800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.926727    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h" event={"ID":"db694a80-da92-4cab-85b5-ed1919c17da6","Type":"ContainerDied","Data":"773ca2a7828738a7c423d7b08fee71fc85036bb9150c1eb98068cdf891925919"}
Nov 04 01:27:06.926800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:06.926773    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773ca2a7828738a7c423d7b08fee71fc85036bb9150c1eb98068cdf891925919"
Nov 04 01:27:07.517235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:07.516278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:07.794754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:07.794661    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:08.037369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.037331    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:08.296534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.296497    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:08.542773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.541508    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:08.638392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.637179    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"]
Nov 04 01:27:08.679375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.673717    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27bc4f8b-jj6td"]
Nov 04 01:27:08.838188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.838152    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"]
Nov 04 01:27:08.838764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.838419    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy" containerID="cri-o://9a54254ca043412ffd80eee661c74f758598b33e74c2d698c512df2dc99286bc" gracePeriod=30
Nov 04 01:27:08.861715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:08.861562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:09.076270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.076121    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:09.183953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.183907    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing" containerID="cri-o://12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb" gracePeriod=25
Nov 04 01:27:09.192449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.192408    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing" containerID="cri-o://f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1" gracePeriod=25
Nov 04 01:27:09.454440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.454404    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:09.666859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.666819    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:09.696372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.696345    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:27:09.780650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.780615    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume\") pod \"a2383056-569f-4805-8ee0-23ee3268483a\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") "
Nov 04 01:27:09.780828 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.780777    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9\" (UniqueName: \"kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9\") pod \"a2383056-569f-4805-8ee0-23ee3268483a\" (UID: \"a2383056-569f-4805-8ee0-23ee3268483a\") "
Nov 04 01:27:09.781094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.781066    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume" pod "a2383056-569f-4805-8ee0-23ee3268483a" (UID: "a2383056-569f-4805-8ee0-23ee3268483a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:09.781800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.781777    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2383056-569f-4805-8ee0-23ee3268483a-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:09.790640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.790606    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9" pod "a2383056-569f-4805-8ee0-23ee3268483a" (UID: "a2383056-569f-4805-8ee0-23ee3268483a"). InnerVolumeSpecName "kube-api-access-4mgv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:09.885865 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.884760    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mgv9\" (UniqueName: \"kubernetes.io/projected/a2383056-569f-4805-8ee0-23ee3268483a-kube-api-access-4mgv9\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:09.965017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:09.963382    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:10.085765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.085630    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:27:10.164094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.161954    5797 generic.go:356] "Generic (PLEG): container finished" podID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerID="12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb" exitCode=0
Nov 04 01:27:10.164094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.162062    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" event={"ID":"e4ef37fa-243b-4c4f-a707-bf1b05287a42","Type":"ContainerDied","Data":"12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb"}
Nov 04 01:27:10.164094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.162384    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m" event={"ID":"e4ef37fa-243b-4c4f-a707-bf1b05287a42","Type":"ContainerDied","Data":"5dfbf063fab4c0cb3cd9d85cc53e87751ca3c5bfdf29eb5599e28ee64745641b"}
Nov 04 01:27:10.164094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.162481    5797 scope.go:122] "RemoveContainer" containerID="12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb"
Nov 04 01:27:10.164094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.162163    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"
Nov 04 01:27:10.165858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.165821    5797 generic.go:356] "Generic (PLEG): container finished" podID="a2383056-569f-4805-8ee0-23ee3268483a" containerID="f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1" exitCode=0
Nov 04 01:27:10.166054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.165936    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" event={"ID":"a2383056-569f-4805-8ee0-23ee3268483a","Type":"ContainerDied","Data":"f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1"}
Nov 04 01:27:10.166054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.165981    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck" event={"ID":"a2383056-569f-4805-8ee0-23ee3268483a","Type":"ContainerDied","Data":"7a156233ec25123799d122c2b811162333b6c03e67225b4c97c0f99514a957e7"}
Nov 04 01:27:10.166217 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.166119    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"
Nov 04 01:27:10.190617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.188731    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh\" (UniqueName: \"kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh\") pod \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") "
Nov 04 01:27:10.190617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.188812    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume\") pod \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\" (UID: \"e4ef37fa-243b-4c4f-a707-bf1b05287a42\") "
Nov 04 01:27:10.196148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.194243    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume" pod "e4ef37fa-243b-4c4f-a707-bf1b05287a42" (UID: "e4ef37fa-243b-4c4f-a707-bf1b05287a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:10.196148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.195443    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:10.204798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.204716    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh" pod "e4ef37fa-243b-4c4f-a707-bf1b05287a42" (UID: "e4ef37fa-243b-4c4f-a707-bf1b05287a42"). InnerVolumeSpecName "kube-api-access-nqkxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:10.211805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.211776    5797 scope.go:122] "RemoveContainer" containerID="12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb"
Nov 04 01:27:10.212340 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:10.212301    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb\": container with ID starting with 12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb not found: ID does not exist" containerID="12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb"
Nov 04 01:27:10.212448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.212354    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb"} err="failed to get container status \"12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb\": rpc error: code = NotFound desc = could not find container \"12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb\": container with ID starting with 12a48011c9e4527b13f10f6e5c1bde35a7d6fae0f4b37d2f15e023c70c9bd4fb not found: ID does not exist"
Nov 04 01:27:10.212551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.212531    5797 scope.go:122] "RemoveContainer" containerID="f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1"
Nov 04 01:27:10.223902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.222561    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"]
Nov 04 01:27:10.241272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.241218    5797 scope.go:122] "RemoveContainer" containerID="f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1"
Nov 04 01:27:10.241704 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:10.241677    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1\": container with ID starting with f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1 not found: ID does not exist" containerID="f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1"
Nov 04 01:27:10.241860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.241715    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1"} err="failed to get container status \"f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1\": rpc error: code = NotFound desc = could not find container \"f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1\": container with ID starting with f9b053893c5965cea2009d9d100e79412f696492b60805f57e005f320cf32ce1 not found: ID does not exist"
Nov 04 01:27:10.245439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.245416    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9902c3-649666867-f4pck"]
Nov 04 01:27:10.295541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.295498    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkxh\" (UniqueName: \"kubernetes.io/projected/e4ef37fa-243b-4c4f-a707-bf1b05287a42-kube-api-access-nqkxh\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:10.295541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.295541    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ef37fa-243b-4c4f-a707-bf1b05287a42-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:10.472067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.469502    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:10.657014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.653107    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2383056-569f-4805-8ee0-23ee3268483a" path="/var/lib/kubelet/pods/a2383056-569f-4805-8ee0-23ee3268483a/volumes"
Nov 04 01:27:10.680357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.680320    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a7775b-1d7e-4c72-9960-7657a192aa1f" path="/var/lib/kubelet/pods/a7a7775b-1d7e-4c72-9960-7657a192aa1f/volumes"
Nov 04 01:27:10.701746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.699223    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"]
Nov 04 01:27:10.709861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.709790    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy" containerID="cri-o://f92686dab3a0e30de9b6d71aa0c2bfd9053874fc8f5824c04075667cb1541fba" gracePeriod=30
Nov 04 01:27:10.710436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.710133    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:10.785959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.785918    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"]
Nov 04 01:27:10.790278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:10.790243    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9902c3-69c8945f86-kd55m"]
Nov 04 01:27:11.367192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:11.367160    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:11.564364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:11.564313    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:11.738590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:11.738496    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:11.738590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:11.738547    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:11.824313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:11.824283    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:12.187618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.187587    5797 generic.go:356] "Generic (PLEG): container finished" podID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerID="275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e" exitCode=0
Nov 04 01:27:12.187862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.187658    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" event={"ID":"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61","Type":"ContainerDied","Data":"275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"}
Nov 04 01:27:12.188216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.188195    5797 scope.go:122] "RemoveContainer" containerID="275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"
Nov 04 01:27:12.189947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.189913    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:27:12.190043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.190028    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:27:12.190099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.190080    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:27:12.404020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.396948    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:12.483486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.483411    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" path="/var/lib/kubelet/pods/e4ef37fa-243b-4c4f-a707-bf1b05287a42/volumes"
Nov 04 01:27:12.751897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:12.751798    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:13.095378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.095299    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"]
Nov 04 01:27:13.095958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.095944    5797 cpu_manager.go:414] "containerMap: removing container" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy"
Nov 04 01:27:13.096029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.095958    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy"
Nov 04 01:27:13.096029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.095977    5797 cpu_manager.go:414] "containerMap: removing container" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy"
Nov 04 01:27:13.096029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.095982    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy"
Nov 04 01:27:13.096029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096008    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing"
Nov 04 01:27:13.096029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096013    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing"
Nov 04 01:27:13.096029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096027    5797 cpu_manager.go:414] "containerMap: removing container" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096033    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096045    5797 cpu_manager.go:414] "containerMap: removing container" podUID="db694a80-da92-4cab-85b5-ed1919c17da6" containerName="revoke-keys"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096049    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db694a80-da92-4cab-85b5-ed1919c17da6" containerName="revoke-keys"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096066    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096071    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096175    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096185    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="59912ca4-fdd9-4f5b-9713-220f3136e515" containerName="istio-proxy"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096193    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ef37fa-243b-4c4f-a707-bf1b05287a42" containerName="payload-processing"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096201    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e762190-ffec-4b55-b757-335f8cadb45a" containerName="istio-proxy"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096206    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="db694a80-da92-4cab-85b5-ed1919c17da6" containerName="revoke-keys"
Nov 04 01:27:13.096227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.096215    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2383056-569f-4805-8ee0-23ee3268483a" containerName="payload-pre-processing"
Nov 04 01:27:13.100300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.100275    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.106908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.106882    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:27:13.110252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.110226    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"]
Nov 04 01:27:13.110495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.110471    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:27:13.203067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.202618    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" event={"ID":"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61","Type":"ContainerStarted","Data":"49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608"}
Nov 04 01:27:13.269022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.266879    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:13.282022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.279166    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.282022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.279566    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pnpn\" (UniqueName: \"kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.282022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.279765    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.381604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.381570    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.381807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.381614    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.381925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.381902    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pnpn\" (UniqueName: \"kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.382395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.382374    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.384314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.384287    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.393924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.393899    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pnpn\" (UniqueName: \"kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn\") pod \"maas-api-revoke-sub-37ea0dd7-t985w\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.417351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.417313    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:13.734950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:13.734921    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"]
Nov 04 01:27:13.737469 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:27:13.737439    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629ae3d0_ccae_4570_b4da_83dbb480ec5e.slice/crio-35d296d6006ba4de6eebee28a4abb0e11c0bf68c62c93472614e9fef07f7c908 WatchSource:0}: Error finding container 35d296d6006ba4de6eebee28a4abb0e11c0bf68c62c93472614e9fef07f7c908: Status 404 returned error can't find the container with id 35d296d6006ba4de6eebee28a4abb0e11c0bf68c62c93472614e9fef07f7c908
Nov 04 01:27:14.004953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.004925    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:27:14.005244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.005215    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:14.005321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.005297    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:14.223774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.222907    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w" event={"ID":"629ae3d0-ccae-4570-b4da-83dbb480ec5e","Type":"ContainerStarted","Data":"35d296d6006ba4de6eebee28a4abb0e11c0bf68c62c93472614e9fef07f7c908"}
Nov 04 01:27:14.227142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.227094    5797 generic.go:356] "Generic (PLEG): container finished" podID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerID="9a54254ca043412ffd80eee661c74f758598b33e74c2d698c512df2dc99286bc" exitCode=0
Nov 04 01:27:14.227454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.227140    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" event={"ID":"b3056bef-bc61-4c70-8aa9-1a0673bf5733","Type":"ContainerDied","Data":"9a54254ca043412ffd80eee661c74f758598b33e74c2d698c512df2dc99286bc"}
Nov 04 01:27:14.229012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.228975    5797 scope.go:122] "RemoveContainer" containerID="f5a4cf00c879b563379c62f69ed751cdf4158c744c2244d4d1cdc547584215bb"
Nov 04 01:27:14.322013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.321958    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:27:14.378699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.358919    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535578    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535752    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535792    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535829    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535879    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535916    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.535964    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.536048    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.536095    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs\") pod \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\" (UID: \"b3056bef-bc61-4c70-8aa9-1a0673bf5733\") "
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.537441    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.537645    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:14.538064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.537821    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:14.592688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.538126    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:14.592688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.539102    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:14.592688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.590669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:14.667148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.667098    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.667148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.667131    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.667148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.667146    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.667373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.667159    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.667373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.667175    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.705013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.668578    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:14.705013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.673202    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "kube-api-access-zv2kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:14.736012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.734847    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:14.744914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.744856    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy" pod "b3056bef-bc61-4c70-8aa9-1a0673bf5733" (UID: "b3056bef-bc61-4c70-8aa9-1a0673bf5733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:14.780496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.780391    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.780496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.780433    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2kp\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-kube-api-access-zv2kp\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.780496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.780451    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:14.780496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:14.780466    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3056bef-bc61-4c70-8aa9-1a0673bf5733-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:15.004815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.004779    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:15.005009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.004850    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:15.241960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.241847    5797 generic.go:356] "Generic (PLEG): container finished" podID="629ae3d0-ccae-4570-b4da-83dbb480ec5e" containerID="6ba3f2968d83cbfd5a8d5dd33b7586d851d0e44ef6ad507f6812dea58592bb34" exitCode=0
Nov 04 01:27:15.241960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.241926    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w" event={"ID":"629ae3d0-ccae-4570-b4da-83dbb480ec5e","Type":"ContainerDied","Data":"6ba3f2968d83cbfd5a8d5dd33b7586d851d0e44ef6ad507f6812dea58592bb34"}
Nov 04 01:27:15.244166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.244145    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"
Nov 04 01:27:15.244316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.244174    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk" event={"ID":"b3056bef-bc61-4c70-8aa9-1a0673bf5733","Type":"ContainerDied","Data":"7caab63d7dc3359ec47c8ecfdb297ce496f71d51da7a355ff8ac9d697c568835"}
Nov 04 01:27:15.244316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.244226    5797 scope.go:122] "RemoveContainer" containerID="9a54254ca043412ffd80eee661c74f758598b33e74c2d698c512df2dc99286bc"
Nov 04 01:27:15.300784 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.300748    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"]
Nov 04 01:27:15.304783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.304754    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27bc4f8b-openshift-default-948b689b9-767dk"]
Nov 04 01:27:15.585610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.585514    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:15.807593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:15.807553    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:16.018115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.018078    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:16.018618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.018585    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:16.277506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.277142    5797 generic.go:356] "Generic (PLEG): container finished" podID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerID="f92686dab3a0e30de9b6d71aa0c2bfd9053874fc8f5824c04075667cb1541fba" exitCode=0
Nov 04 01:27:16.277982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.277819    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" event={"ID":"33f3f13f-1942-4dca-b993-0f48deaa503e","Type":"ContainerDied","Data":"f92686dab3a0e30de9b6d71aa0c2bfd9053874fc8f5824c04075667cb1541fba"}
Nov 04 01:27:16.277982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.277883    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz" event={"ID":"33f3f13f-1942-4dca-b993-0f48deaa503e","Type":"ContainerDied","Data":"af22ad3c60d50c2f41cceff6dd15a813fc63384e4855b14ec8bd42798d3b42eb"}
Nov 04 01:27:16.277982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.277910    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af22ad3c60d50c2f41cceff6dd15a813fc63384e4855b14ec8bd42798d3b42eb"
Nov 04 01:27:16.277982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.277929    5797 scope.go:122] "RemoveContainer" containerID="9c02dd011ff8f37cdc4a2431185502d486e3d8d2b0cc5c4c685bd627b3bf06f3"
Nov 04 01:27:16.292021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.291996    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:27:16.438428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.438300    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:16.485065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.485032    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492152    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492282    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492323    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492347    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492398    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492441    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492532    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.492758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.492705    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data\") pod \"33f3f13f-1942-4dca-b993-0f48deaa503e\" (UID: \"33f3f13f-1942-4dca-b993-0f48deaa503e\") "
Nov 04 01:27:16.499929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.485906    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:16.499929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.496617    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:16.499929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.499751    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "kube-api-access-n6vf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:16.500234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.500073    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:16.508030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.504122    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:16.508030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.506474    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:16.514661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.511433    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:16.514661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.512824    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:16.514661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.513874    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" path="/var/lib/kubelet/pods/b3056bef-bc61-4c70-8aa9-1a0673bf5733/volumes"
Nov 04 01:27:16.516848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.516826    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy" pod "33f3f13f-1942-4dca-b993-0f48deaa503e" (UID: "33f3f13f-1942-4dca-b993-0f48deaa503e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:16.596683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.596646    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f3f13f-1942-4dca-b993-0f48deaa503e-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597219    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597347    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597454    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597548    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597643    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vf5\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-kube-api-access-n6vf5\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597733    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597820    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.597923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.597912    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f3f13f-1942-4dca-b993-0f48deaa503e-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.689499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.689415    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:16.801175 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.801137    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca\") pod \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") "
Nov 04 01:27:16.801396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.801210    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn\" (UniqueName: \"kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn\") pod \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") "
Nov 04 01:27:16.801461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.801396    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca\") pod \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\" (UID: \"629ae3d0-ccae-4570-b4da-83dbb480ec5e\") "
Nov 04 01:27:16.801574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.801546    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca" pod "629ae3d0-ccae-4570-b4da-83dbb480ec5e" (UID: "629ae3d0-ccae-4570-b4da-83dbb480ec5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:16.802151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.802129    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.803487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.803451    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca" pod "629ae3d0-ccae-4570-b4da-83dbb480ec5e" (UID: "629ae3d0-ccae-4570-b4da-83dbb480ec5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:16.803677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.803664    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn" pod "629ae3d0-ccae-4570-b4da-83dbb480ec5e" (UID: "629ae3d0-ccae-4570-b4da-83dbb480ec5e"). InnerVolumeSpecName "kube-api-access-6pnpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:16.913074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.913019    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/629ae3d0-ccae-4570-b4da-83dbb480ec5e-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:16.913074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:16.913056    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pnpn\" (UniqueName: \"kubernetes.io/projected/629ae3d0-ccae-4570-b4da-83dbb480ec5e-kube-api-access-6pnpn\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:17.004941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.004862    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:17.004941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.004917    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:17.075124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.075082    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:17.295390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.295297    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w" event={"ID":"629ae3d0-ccae-4570-b4da-83dbb480ec5e","Type":"ContainerDied","Data":"35d296d6006ba4de6eebee28a4abb0e11c0bf68c62c93472614e9fef07f7c908"}
Nov 04 01:27:17.295575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.295401    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d296d6006ba4de6eebee28a4abb0e11c0bf68c62c93472614e9fef07f7c908"
Nov 04 01:27:17.295575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.295486    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"
Nov 04 01:27:17.300487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.299857    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"
Nov 04 01:27:17.326701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.325516    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:17.386010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.385958    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"]
Nov 04 01:27:17.404884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.404845    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fdd516-openshift-default-6844bbb85f-hngzz"]
Nov 04 01:27:17.539010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.536614    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:17.652081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.649399    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"]
Nov 04 01:27:17.819026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:17.816364    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:18.004981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.004514    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:18.004981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.004572    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:18.130012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.127500    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:18.366078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.347368    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:18.484318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.484280    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" path="/var/lib/kubelet/pods/33f3f13f-1942-4dca-b993-0f48deaa503e/volumes"
Nov 04 01:27:18.704150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.703387    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:18.942704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.942665    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"]
Nov 04 01:27:18.942971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.942937    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" containerID="cri-o://e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af" gracePeriod=30
Nov 04 01:27:18.984422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.984294    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"]
Nov 04 01:27:18.985907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:18.984661    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" containerID="cri-o://dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503" gracePeriod=30
Nov 04 01:27:19.005361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:19.005325    5797 patch_prober.go:29] interesting pod/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 04 01:27:19.005531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:19.005376    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 04 01:27:19.071515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:19.071479    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:19.541539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:19.541491    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:19.748336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:19.748297    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:20.177011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:20.176259    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:27:20.178060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:20.178025    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:27:20.179798 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:20.179757    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:27:20.258937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:20.258889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:20.758106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:20.741376    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:20.968065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:20.957053    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"]
Nov 04 01:27:21.234155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.234119    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:21.442766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.442709    5797 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" secret="" err="secret \"e2e-api-b-20a250-openshift-default-dockercfg-qqxws\" not found"
Nov 04 01:27:21.444598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.444517    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:21.637218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.637180    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"]
Nov 04 01:27:21.638520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.638490    5797 cpu_manager.go:414] "containerMap: removing container" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy"
Nov 04 01:27:21.638672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.638657    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy"
Nov 04 01:27:21.638801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.638788    5797 cpu_manager.go:414] "containerMap: removing container" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy"
Nov 04 01:27:21.638894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.638882    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy"
Nov 04 01:27:21.639017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.638982    5797 cpu_manager.go:414] "containerMap: removing container" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy"
Nov 04 01:27:21.639142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639129    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy"
Nov 04 01:27:21.639243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639233    5797 cpu_manager.go:414] "containerMap: removing container" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy"
Nov 04 01:27:21.639332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639319    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy"
Nov 04 01:27:21.639428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639417    5797 cpu_manager.go:414] "containerMap: removing container" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy"
Nov 04 01:27:21.639506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639495    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy"
Nov 04 01:27:21.639691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639676    5797 cpu_manager.go:414] "containerMap: removing container" podUID="629ae3d0-ccae-4570-b4da-83dbb480ec5e" containerName="revoke-keys"
Nov 04 01:27:21.639794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.639781    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629ae3d0-ccae-4570-b4da-83dbb480ec5e" containerName="revoke-keys"
Nov 04 01:27:21.640164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.640146    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="629ae3d0-ccae-4570-b4da-83dbb480ec5e" containerName="revoke-keys"
Nov 04 01:27:21.640315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.640301    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy"
Nov 04 01:27:21.640413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.640402    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy"
Nov 04 01:27:21.640503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.640492    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f3f13f-1942-4dca-b993-0f48deaa503e" containerName="istio-proxy"
Nov 04 01:27:21.640632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.640618    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3056bef-bc61-4c70-8aa9-1a0673bf5733" containerName="istio-proxy"
Nov 04 01:27:21.640733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.640722    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="af20f3c0-5693-4ff4-9587-616247f7553e" containerName="istio-proxy"
Nov 04 01:27:21.667015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.660135    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"]
Nov 04 01:27:21.667015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.660227    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.709310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.709269    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:27:21.709511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.709419    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:27:21.796017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.790837    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.796017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.790891    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.796017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.790939    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8rg\" (UniqueName: \"kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.898103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.897088    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.898103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.897148    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.898103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.897183    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8rg\" (UniqueName: \"kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.950509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.920498    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.950509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.941532    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8rg\" (UniqueName: \"kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:21.968393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.968352    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"]
Nov 04 01:27:21.981287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:21.975294    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfed358-xf5rh\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:22.017014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.016286    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"]
Nov 04 01:27:22.017014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.016455    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.045050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.036394    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:22.298330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.272624    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.298330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.272818    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.298330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.272867    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9x5r\" (UniqueName: \"kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.433944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.404162    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.433944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.404394    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.433944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.404451    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9x5r\" (UniqueName: \"kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.433944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.405876    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.492156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.482698    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9x5r\" (UniqueName: \"kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.576805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.576721    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af3d9893-mhlmk\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:22.617220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.617177    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy" containerID="cri-o://49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608" gracePeriod=30
Nov 04 01:27:22.776592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:22.776542    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:23.225067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.123747    5797 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.225067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.211980    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.285010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.281352    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.400433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.400393    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 01:27:23.497034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.496924    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.516541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.516491    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.558533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.543197    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.632152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.630927    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 04 01:27:23.634970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.634933    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"]
Nov 04 01:27:23.828133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.828084    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:27:23.948556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:23.844224    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:24.509413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.508724    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 04 01:27:24.509413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.508793    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 04 01:27:24.530160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.530094    5797 patch_prober.go:29] interesting pod/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 04 01:27:24.530160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.530152    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 04 01:27:24.555199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.555156    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"]
Nov 04 01:27:24.732390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.732364    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798440    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798585    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798641    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798741    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798810    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798843    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.798947    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.799080    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.799187    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy\") pod \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\" (UID: \"e7af977b-c72f-4da5-81b1-e5eb0de7ead0\") "
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.800843    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.801209    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.801315    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.801567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.801434    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.802614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.802543    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.853869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.853804    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.853869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.853815    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "kube-api-access-h5465". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:24.853869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.853810    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:24.854394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.854366    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token" pod "e7af977b-c72f-4da5-81b1-e5eb0de7ead0" (UID: "e7af977b-c72f-4da5-81b1-e5eb0de7ead0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:24.871041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.871016    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:27:24.900648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.900617    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.900833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.900677    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.900833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.900754    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.900833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.900790    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.901052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.900881    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.901052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.900936    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.901265 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.901058    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:24.901265 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.901219    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.901422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.901305    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.901422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.901339    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket\") pod \"a6b7fa46-feb6-4b67-895a-9624c2511f00\" (UID: \"a6b7fa46-feb6-4b67-895a-9624c2511f00\") "
Nov 04 01:27:24.902152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902125    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5465\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-kube-api-access-h5465\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902155    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7fa46-feb6-4b67-895a-9624c2511f00-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902168    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902182    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902193    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902206    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902219    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902231    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902246    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.902295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.902267    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7af977b-c72f-4da5-81b1-e5eb0de7ead0-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:24.905704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.904546    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.905704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.904642    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.905704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.904665    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.905704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.904696    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.908136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.908089    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:24.908252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.908165    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:24.912087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.912056    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:24.912087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:24.912061    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6" pod "a6b7fa46-feb6-4b67-895a-9624c2511f00" (UID: "a6b7fa46-feb6-4b67-895a-9624c2511f00"). InnerVolumeSpecName "kube-api-access-z92n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003221    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z92n6\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-kube-api-access-z92n6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003255    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003266    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003277    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003286    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003294    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b7fa46-feb6-4b67-895a-9624c2511f00-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003302    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.003309 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.003311    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b7fa46-feb6-4b67-895a-9624c2511f00-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:25.083635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.083581    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:25.243182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.243146    5797 generic.go:356] "Generic (PLEG): container finished" podID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerID="dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503" exitCode=0
Nov 04 01:27:25.243409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.243232    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" event={"ID":"a6b7fa46-feb6-4b67-895a-9624c2511f00","Type":"ContainerDied","Data":"dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503"}
Nov 04 01:27:25.243409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.243223    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"
Nov 04 01:27:25.243409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.243327    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg" event={"ID":"a6b7fa46-feb6-4b67-895a-9624c2511f00","Type":"ContainerDied","Data":"a9b1eba6ea518434a7f2e046e8e91c0303f0ef60cb3b605670f005736c486310"}
Nov 04 01:27:25.243409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.243360    5797 scope.go:122] "RemoveContainer" containerID="dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503"
Nov 04 01:27:25.245721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.245435    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh" event={"ID":"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442","Type":"ContainerStarted","Data":"769f2402a588c1a509f6172a6cfc5a57df7746ec285dcce72363d130056bf652"}
Nov 04 01:27:25.245721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.245503    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh" event={"ID":"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442","Type":"ContainerStarted","Data":"9b6f30d7de0d9af9e16cdc8177b4297402230224f33c87a739b7e2337d2c0794"}
Nov 04 01:27:25.247303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.247276    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk" event={"ID":"2a135510-23e8-4e79-a77c-42391aa008a3","Type":"ContainerStarted","Data":"d2ab4a3ea258678cda3de74bc384584f59b17df44cc0a3c687e0fec86b8e365c"}
Nov 04 01:27:25.247484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.247323    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk" event={"ID":"2a135510-23e8-4e79-a77c-42391aa008a3","Type":"ContainerStarted","Data":"6ef91580caaf87943a327b812e86463a848a8498fa6dc0f884d2995a89f5f5dc"}
Nov 04 01:27:25.250778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.250755    5797 generic.go:356] "Generic (PLEG): container finished" podID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerID="e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af" exitCode=0
Nov 04 01:27:25.251373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.250934    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" event={"ID":"e7af977b-c72f-4da5-81b1-e5eb0de7ead0","Type":"ContainerDied","Data":"e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af"}
Nov 04 01:27:25.251373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.250979    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr" event={"ID":"e7af977b-c72f-4da5-81b1-e5eb0de7ead0","Type":"ContainerDied","Data":"c176b920f0625091d41954bfc242d549685e3e10084f9f0f414b2356a989e31f"}
Nov 04 01:27:25.251373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.251099    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"
Nov 04 01:27:25.258667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.258642    5797 scope.go:122] "RemoveContainer" containerID="dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503"
Nov 04 01:27:25.258946 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:25.258926    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503\": container with ID starting with dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503 not found: ID does not exist" containerID="dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503"
Nov 04 01:27:25.259055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.258959    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503"} err="failed to get container status \"dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503\": rpc error: code = NotFound desc = could not find container \"dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503\": container with ID starting with dec1fac730f85015c6ee291eec09abb65149e3f5bfcc52475f405e425fe66503 not found: ID does not exist"
Nov 04 01:27:25.259055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.258983    5797 scope.go:122] "RemoveContainer" containerID="e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af"
Nov 04 01:27:25.266898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.266856    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk" podStartSLOduration=4.26684564 podStartE2EDuration="4.26684564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:25.265636213 +0000 UTC m=+1432.907542498" watchObservedRunningTime="2026-11-04 01:27:25.26684564 +0000 UTC m=+1432.908751890"
Nov 04 01:27:25.270200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.270178    5797 scope.go:122] "RemoveContainer" containerID="e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af"
Nov 04 01:27:25.270541 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:25.270520    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af\": container with ID starting with e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af not found: ID does not exist" containerID="e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af"
Nov 04 01:27:25.270637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.270549    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af"} err="failed to get container status \"e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af\": rpc error: code = NotFound desc = could not find container \"e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af\": container with ID starting with e29dfa19f81060ea18ae42da8a45f2e1ec5dc33ab79f3ec7f8818eeb6c4bf7af not found: ID does not exist"
Nov 04 01:27:25.283728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.283680    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh" podStartSLOduration=4.283662741 podStartE2EDuration="4.283662741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:25.283239701 +0000 UTC m=+1432.925145974" watchObservedRunningTime="2026-11-04 01:27:25.283662741 +0000 UTC m=+1432.925568993"
Nov 04 01:27:25.287269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.287242    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:25.301424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.301397    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"]
Nov 04 01:27:25.306683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.306657    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-nrgbr"]
Nov 04 01:27:25.329106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.329076    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"]
Nov 04 01:27:25.334416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.334387    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9902c3-openshift-default-58c9d957d5-fslsg"]
Nov 04 01:27:25.931471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:25.931394    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:26.093973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.093886    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"]
Nov 04 01:27:26.112749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.112719    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"]
Nov 04 01:27:26.256197 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:26.256164    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:27:26.263373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.263095    5797 generic.go:356] "Generic (PLEG): container finished" podID="2a135510-23e8-4e79-a77c-42391aa008a3" containerID="d2ab4a3ea258678cda3de74bc384584f59b17df44cc0a3c687e0fec86b8e365c" exitCode=0
Nov 04 01:27:26.263373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.263128    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk" event={"ID":"2a135510-23e8-4e79-a77c-42391aa008a3","Type":"ContainerDied","Data":"d2ab4a3ea258678cda3de74bc384584f59b17df44cc0a3c687e0fec86b8e365c"}
Nov 04 01:27:26.267606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.267580    5797 generic.go:356] "Generic (PLEG): container finished" podID="7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" containerID="769f2402a588c1a509f6172a6cfc5a57df7746ec285dcce72363d130056bf652" exitCode=0
Nov 04 01:27:26.267746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.267658    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh" event={"ID":"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442","Type":"ContainerDied","Data":"769f2402a588c1a509f6172a6cfc5a57df7746ec285dcce72363d130056bf652"}
Nov 04 01:27:26.287684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.287621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:26.480392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.480356    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" path="/var/lib/kubelet/pods/a6b7fa46-feb6-4b67-895a-9624c2511f00/volumes"
Nov 04 01:27:26.480970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.480952    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" path="/var/lib/kubelet/pods/e7af977b-c72f-4da5-81b1-e5eb0de7ead0/volumes"
Nov 04 01:27:26.894700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:26.894670    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:27.252789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.252702    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"]
Nov 04 01:27:27.253251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.252950    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy" containerID="cri-o://754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1" gracePeriod=30
Nov 04 01:27:27.305050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.293088    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"]
Nov 04 01:27:27.305050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.293457    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy" containerID="cri-o://4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22" gracePeriod=30
Nov 04 01:27:27.563911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.563880    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:27.566681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.566654    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:27.708013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.695277    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca\") pod \"2a135510-23e8-4e79-a77c-42391aa008a3\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") "
Nov 04 01:27:27.708013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.703396    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca\") pod \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") "
Nov 04 01:27:27.708013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.703574    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca\") pod \"2a135510-23e8-4e79-a77c-42391aa008a3\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") "
Nov 04 01:27:27.738013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.703218    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca" pod "2a135510-23e8-4e79-a77c-42391aa008a3" (UID: "2a135510-23e8-4e79-a77c-42391aa008a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:27.738013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.714910    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca\") pod \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") "
Nov 04 01:27:27.738013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.714960    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg\" (UniqueName: \"kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg\") pod \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\" (UID: \"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442\") "
Nov 04 01:27:27.738013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.715074    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r\" (UniqueName: \"kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r\") pod \"2a135510-23e8-4e79-a77c-42391aa008a3\" (UID: \"2a135510-23e8-4e79-a77c-42391aa008a3\") "
Nov 04 01:27:27.738013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.715646    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:27.738013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.716634    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca" pod "7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" (UID: "7aba32a2-caa6-4015-8aa0-a1ae9c5ad442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:27.740516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.740477    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg" pod "7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" (UID: "7aba32a2-caa6-4015-8aa0-a1ae9c5ad442"). InnerVolumeSpecName "kube-api-access-bx8rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:27.750015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.749697    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca" pod "2a135510-23e8-4e79-a77c-42391aa008a3" (UID: "2a135510-23e8-4e79-a77c-42391aa008a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:27.762423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.762166    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca" pod "7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" (UID: "7aba32a2-caa6-4015-8aa0-a1ae9c5ad442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:27.834032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.830667    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a135510-23e8-4e79-a77c-42391aa008a3-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:27.834032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.830708    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:27.834032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.830724    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8rg\" (UniqueName: \"kubernetes.io/projected/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-kube-api-access-bx8rg\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:27.834032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.830739    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:27.834391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.834158    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r" pod "2a135510-23e8-4e79-a77c-42391aa008a3" (UID: "2a135510-23e8-4e79-a77c-42391aa008a3"). InnerVolumeSpecName "kube-api-access-b9x5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:27.931903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.931829    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9x5r\" (UniqueName: \"kubernetes.io/projected/2a135510-23e8-4e79-a77c-42391aa008a3-kube-api-access-b9x5r\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:27.987329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:27.987290    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:28.007499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.007470    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:27:28.142159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142123    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.142334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142174    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.142334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142206    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.142334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142227    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.142334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142276    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142422    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142529    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142640    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142645    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142658    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142727    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142761    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.142775    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket\") pod \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\" (UID: \"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61\") "
Nov 04 01:27:28.143434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.143046    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:28.143933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.143565    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.143933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.143588    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.143933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.143604    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.143933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.143619    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.143933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.143634    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.144939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.144913    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "kube-api-access-fzb2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:28.144939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.144926    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:28.145126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.145018    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:28.145126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.145070    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo" pod "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" (UID: "5f9ad66a-bede-4e9a-94ea-f9dd9a924f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:28.193428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.193387    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:28.245429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.245392    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.245429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.245423    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzb2q\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-kube-api-access-fzb2q\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.245429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.245433    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.245689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.245442    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:28.295473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.295444    5797 generic.go:356] "Generic (PLEG): container finished" podID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerID="49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608" exitCode=0
Nov 04 01:27:28.295918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.295495    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"
Nov 04 01:27:28.295918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.295528    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" event={"ID":"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61","Type":"ContainerDied","Data":"49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608"}
Nov 04 01:27:28.295918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.295584    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2" event={"ID":"5f9ad66a-bede-4e9a-94ea-f9dd9a924f61","Type":"ContainerDied","Data":"861fc64134ffaa424734c20a32822a3ed98687e1a2ad9852c4b3a8941f944214"}
Nov 04 01:27:28.295918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.295611    5797 scope.go:122] "RemoveContainer" containerID="49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608"
Nov 04 01:27:28.301345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.301271    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"
Nov 04 01:27:28.301345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.301291    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh" event={"ID":"7aba32a2-caa6-4015-8aa0-a1ae9c5ad442","Type":"ContainerDied","Data":"9b6f30d7de0d9af9e16cdc8177b4297402230224f33c87a739b7e2337d2c0794"}
Nov 04 01:27:28.301561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.301359    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6f30d7de0d9af9e16cdc8177b4297402230224f33c87a739b7e2337d2c0794"
Nov 04 01:27:28.302948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.302922    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk" event={"ID":"2a135510-23e8-4e79-a77c-42391aa008a3","Type":"ContainerDied","Data":"6ef91580caaf87943a327b812e86463a848a8498fa6dc0f884d2995a89f5f5dc"}
Nov 04 01:27:28.303098 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.302957    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ef91580caaf87943a327b812e86463a848a8498fa6dc0f884d2995a89f5f5dc"
Nov 04 01:27:28.303098 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.303046    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"
Nov 04 01:27:28.308584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.308563    5797 scope.go:122] "RemoveContainer" containerID="275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"
Nov 04 01:27:28.321109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.321091    5797 scope.go:122] "RemoveContainer" containerID="49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608"
Nov 04 01:27:28.321402 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:28.321381    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608\": container with ID starting with 49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608 not found: ID does not exist" containerID="49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608"
Nov 04 01:27:28.321452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.321412    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608"} err="failed to get container status \"49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608\": rpc error: code = NotFound desc = could not find container \"49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608\": container with ID starting with 49b78bfc871ee4258333490f58c6fff4d40a3dcf27c5176aac85516033c65608 not found: ID does not exist"
Nov 04 01:27:28.321452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.321430    5797 scope.go:122] "RemoveContainer" containerID="275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"
Nov 04 01:27:28.321681 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:28.321663    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e\": container with ID starting with 275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e not found: ID does not exist" containerID="275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"
Nov 04 01:27:28.321801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.321685    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e"} err="failed to get container status \"275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e\": rpc error: code = NotFound desc = could not find container \"275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e\": container with ID starting with 275b5c82a7efdfc9e2d11b3837089c80af89f491c53c24b83a35c9c04da8229e not found: ID does not exist"
Nov 04 01:27:28.333627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.333579    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"]
Nov 04 01:27:28.338316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.338290    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-20a250-openshift-default-6f5846d95f-pbcp2"]
Nov 04 01:27:28.479889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.479811    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" path="/var/lib/kubelet/pods/5f9ad66a-bede-4e9a-94ea-f9dd9a924f61/volumes"
Nov 04 01:27:28.529200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.529166    5797 scope.go:122] "RemoveContainer" containerID="d3c63f2aa922549a146c2b064dcab074ddd97d6b1dfde8db1f17fcf9cb7c38a5"
Nov 04 01:27:28.541243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.541223    5797 scope.go:122] "RemoveContainer" containerID="b3e2aa22ac64b672ec3f91e951b78ea849992351666acd99b570c7f81f4d10d7"
Nov 04 01:27:28.556133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.556105    5797 scope.go:122] "RemoveContainer" containerID="5caac918a86fee74bd10089ac0415eecfad4148d576e966496ce4fdf253b79e1"
Nov 04 01:27:28.872247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:28.872217    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:29.175358 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:29.175263    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:29.372764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:29.372722    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:29.643622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:29.643588    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:29.957834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:29.957736    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:30.188016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:30.187823    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:30.403976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:30.403939    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:30.632016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:30.629018    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:30.851477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:30.851385    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:31.395632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:31.395594    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:32.008294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.008265    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:32.224402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.223062    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:32.343594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.343555    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"]
Nov 04 01:27:32.344666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344590    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy"
Nov 04 01:27:32.344666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344610    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy"
Nov 04 01:27:32.344666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344648    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" containerName="revoke-keys"
Nov 04 01:27:32.344666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344656    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" containerName="revoke-keys"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344694    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344703    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344716    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344724    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344737    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2a135510-23e8-4e79-a77c-42391aa008a3" containerName="revoke-keys"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344744    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a135510-23e8-4e79-a77c-42391aa008a3" containerName="revoke-keys"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344872    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344888    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b7fa46-feb6-4b67-895a-9624c2511f00" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344900    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7af977b-c72f-4da5-81b1-e5eb0de7ead0" containerName="istio-proxy"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344912    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a135510-23e8-4e79-a77c-42391aa008a3" containerName="revoke-keys"
Nov 04 01:27:32.345574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.344932    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" containerName="revoke-keys"
Nov 04 01:27:32.351780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.351743    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.368653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.359341    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"]
Nov 04 01:27:32.398022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.394048    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:27:32.398022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.394139    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:27:32.436913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.436342    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.436913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.436389    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.436913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.436418    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.545046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.540965    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.545046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.541032    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.545046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.541061    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.545046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.543434    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.545046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.544543    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.559511 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.559480    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8\") pod \"maas-api-revoke-sub-775fb521-fwt6x\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.561365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.561268    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:32.626270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.626233    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:27:32.745086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745051    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745194    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745232    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745248    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745281    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745312    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745446    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745467    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.745536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.745522    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy\") pod \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\" (UID: \"9c7f34d0-b8a1-418f-895b-3a5b61c57bda\") "
Nov 04 01:27:32.746547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.746490    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:32.746649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.746540    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:32.746649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.746628    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:32.747709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.747688    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:32.747904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.747884    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:32.748463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.748437    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:32.749011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.748968    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "kube-api-access-4h4qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:32.749501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.749481    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:32.750092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.750075    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy" pod "9c7f34d0-b8a1-418f-895b-3a5b61c57bda" (UID: "9c7f34d0-b8a1-418f-895b-3a5b61c57bda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:32.775023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.773802    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:32.799728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.799418    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:32.854819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854784    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.854819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854818    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854833    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854861    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854874    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854885    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854899    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854911    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.855084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.854934    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4qv\" (UniqueName: \"kubernetes.io/projected/9c7f34d0-b8a1-418f-895b-3a5b61c57bda-kube-api-access-4h4qv\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:32.956862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:32.956686    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" containerID="cri-o://0b6e7ab5915c45ae63427ef154ac1ae2bed0e79c86f35d01a349b5aacf7739db" gracePeriod=45
Nov 04 01:27:33.012886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.011782    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:33.322967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.322857    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Nov 04 01:27:33.323625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.323596    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"]
Nov 04 01:27:33.324202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.324133    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:27:33.325596 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:27:33.325568    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2fd8ca_d920_4e1e_bc1a_6f1c35513f40.slice/crio-969e913585b92f31240bbfe297efa39c0b470b13f65e6e5a449f2a2e3ce14e8c WatchSource:0}: Error finding container 969e913585b92f31240bbfe297efa39c0b470b13f65e6e5a449f2a2e3ce14e8c: Status 404 returned error can't find the container with id 969e913585b92f31240bbfe297efa39c0b470b13f65e6e5a449f2a2e3ce14e8c
Nov 04 01:27:33.423035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.422634    5797 generic.go:356] "Generic (PLEG): container finished" podID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerID="4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22" exitCode=0
Nov 04 01:27:33.423035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.422785    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"
Nov 04 01:27:33.423249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.423189    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" event={"ID":"66bc8ba2-e063-4d0e-b12d-7a73718366c7","Type":"ContainerDied","Data":"4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22"}
Nov 04 01:27:33.423249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.423242    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn" event={"ID":"66bc8ba2-e063-4d0e-b12d-7a73718366c7","Type":"ContainerDied","Data":"5cbf5b84402ee3a632746d7c3093c1d47903f4d8477c1ae7e9edabccd8d12641"}
Nov 04 01:27:33.423323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.423275    5797 scope.go:122] "RemoveContainer" containerID="4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22"
Nov 04 01:27:33.426834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.426502    5797 generic.go:356] "Generic (PLEG): container finished" podID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerID="754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1" exitCode=0
Nov 04 01:27:33.426834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.426599    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" event={"ID":"9c7f34d0-b8a1-418f-895b-3a5b61c57bda","Type":"ContainerDied","Data":"754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1"}
Nov 04 01:27:33.426834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.426577    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"
Nov 04 01:27:33.426834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.426643    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2" event={"ID":"9c7f34d0-b8a1-418f-895b-3a5b61c57bda","Type":"ContainerDied","Data":"a25a35ea866b953ee1f123866d88357746e1164b802b518b37e7ec450ecdae73"}
Nov 04 01:27:33.429585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.429541    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x" event={"ID":"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40","Type":"ContainerStarted","Data":"969e913585b92f31240bbfe297efa39c0b470b13f65e6e5a449f2a2e3ce14e8c"}
Nov 04 01:27:33.431922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.431894    5797 generic.go:356] "Generic (PLEG): container finished" podID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerID="0b6e7ab5915c45ae63427ef154ac1ae2bed0e79c86f35d01a349b5aacf7739db" exitCode=0
Nov 04 01:27:33.432079 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.432060    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" event={"ID":"eb767966-ca5f-49cd-80aa-1728469ea3b1","Type":"ContainerDied","Data":"0b6e7ab5915c45ae63427ef154ac1ae2bed0e79c86f35d01a349b5aacf7739db"}
Nov 04 01:27:33.464824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.464789    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.464878    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.464949    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.465091    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.465201    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.465261    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.465409    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.465467    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.465545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.465523    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket\") pod \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\" (UID: \"66bc8ba2-e063-4d0e-b12d-7a73718366c7\") "
Nov 04 01:27:33.466543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.466515    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.467092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.467036    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.467202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.467094    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.467560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.467534    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:33.467951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.467922    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.468632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.468538    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.468632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.468564    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.468632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.468580    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.468632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.468592    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.468632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.468605    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.550694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.550649    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.552604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.552566    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:33.552783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.552758    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "kube-api-access-6jf2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:33.553877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.553845    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo" pod "66bc8ba2-e063-4d0e-b12d-7a73718366c7" (UID: "66bc8ba2-e063-4d0e-b12d-7a73718366c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:27:33.570189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.570151    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jf2l\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-kube-api-access-6jf2l\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.570189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.570187    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.570398 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.570203    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.570398 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.570216    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bc8ba2-e063-4d0e-b12d-7a73718366c7-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.596187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.596139    5797 scope.go:122] "RemoveContainer" containerID="4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22"
Nov 04 01:27:33.596504 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:33.596479    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22\": container with ID starting with 4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22 not found: ID does not exist" containerID="4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22"
Nov 04 01:27:33.596592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.596511    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22"} err="failed to get container status \"4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22\": rpc error: code = NotFound desc = could not find container \"4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22\": container with ID starting with 4112404a22b7c2f66dbb30e14a6d8fccc714eebc0a8be02fa9ccab74934f0f22 not found: ID does not exist"
Nov 04 01:27:33.596592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.596529    5797 scope.go:122] "RemoveContainer" containerID="754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1"
Nov 04 01:27:33.613500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.613472    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:27:33.616955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.616933    5797 scope.go:122] "RemoveContainer" containerID="754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1"
Nov 04 01:27:33.617262 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:33.617234    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1\": container with ID starting with 754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1 not found: ID does not exist" containerID="754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1"
Nov 04 01:27:33.617320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.617271    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1"} err="failed to get container status \"754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1\": rpc error: code = NotFound desc = could not find container \"754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1\": container with ID starting with 754db2536c01517ee983f37b9170d24b02327b20a353d4e958b3950398c70eb1 not found: ID does not exist"
Nov 04 01:27:33.627983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.627953    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"]
Nov 04 01:27:33.631508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.631485    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-vw2k2"]
Nov 04 01:27:33.758206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.758178    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"]
Nov 04 01:27:33.764320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.764285    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-xwrpn"]
Nov 04 01:27:33.786571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.786534    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm\") pod \"eb767966-ca5f-49cd-80aa-1728469ea3b1\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") "
Nov 04 01:27:33.786833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.786611    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache\") pod \"eb767966-ca5f-49cd-80aa-1728469ea3b1\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") "
Nov 04 01:27:33.786833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.786797    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9\" (UniqueName: \"kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9\") pod \"eb767966-ca5f-49cd-80aa-1728469ea3b1\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") "
Nov 04 01:27:33.787032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.786877    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home\") pod \"eb767966-ca5f-49cd-80aa-1728469ea3b1\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") "
Nov 04 01:27:33.787032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.786928    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir\") pod \"eb767966-ca5f-49cd-80aa-1728469ea3b1\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") "
Nov 04 01:27:33.787032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.787022    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs\") pod \"eb767966-ca5f-49cd-80aa-1728469ea3b1\" (UID: \"eb767966-ca5f-49cd-80aa-1728469ea3b1\") "
Nov 04 01:27:33.788333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.788215    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir" pod "eb767966-ca5f-49cd-80aa-1728469ea3b1" (UID: "eb767966-ca5f-49cd-80aa-1728469ea3b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.788333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.788271    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache" pod "eb767966-ca5f-49cd-80aa-1728469ea3b1" (UID: "eb767966-ca5f-49cd-80aa-1728469ea3b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.788508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.788463    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.788508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.788488    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.788895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.788869    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home" pod "eb767966-ca5f-49cd-80aa-1728469ea3b1" (UID: "eb767966-ca5f-49cd-80aa-1728469ea3b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.790086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.790059    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9" pod "eb767966-ca5f-49cd-80aa-1728469ea3b1" (UID: "eb767966-ca5f-49cd-80aa-1728469ea3b1"). InnerVolumeSpecName "kube-api-access-mdvw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:33.796494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.796468    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm" pod "eb767966-ca5f-49cd-80aa-1728469ea3b1" (UID: "eb767966-ca5f-49cd-80aa-1728469ea3b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:33.798289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.798250    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs" pod "eb767966-ca5f-49cd-80aa-1728469ea3b1" (UID: "eb767966-ca5f-49cd-80aa-1728469ea3b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:33.890676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.890637    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.890676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.890671    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvw9\" (UniqueName: \"kubernetes.io/projected/eb767966-ca5f-49cd-80aa-1728469ea3b1-kube-api-access-mdvw9\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.890934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.890686    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb767966-ca5f-49cd-80aa-1728469ea3b1-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.890934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.890701    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb767966-ca5f-49cd-80aa-1728469ea3b1-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:33.891381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:33.891348    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:34.103115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.102947    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:34.318357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.318321    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:34.441004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.440971    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"
Nov 04 01:27:34.441166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.441044    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j" event={"ID":"eb767966-ca5f-49cd-80aa-1728469ea3b1","Type":"ContainerDied","Data":"d1c8b29e61bd73418ef4d0ef6b3098bc9dac6d3e720363b36bc11ffc16f30bc9"}
Nov 04 01:27:34.441166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.441099    5797 scope.go:122] "RemoveContainer" containerID="0b6e7ab5915c45ae63427ef154ac1ae2bed0e79c86f35d01a349b5aacf7739db"
Nov 04 01:27:34.445346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.445316    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x" event={"ID":"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40","Type":"ContainerStarted","Data":"d51237da76354521c89e1d51bb3d3de947f0f07043ea017403b429338adc3a58"}
Nov 04 01:27:34.473718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.473658    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x" podStartSLOduration=2.473643172 podStartE2EDuration="2.473643172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:34.472330894 +0000 UTC m=+1442.114237144" watchObservedRunningTime="2026-11-04 01:27:34.473643172 +0000 UTC m=+1442.115549448"
Nov 04 01:27:34.482622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.482591    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" path="/var/lib/kubelet/pods/66bc8ba2-e063-4d0e-b12d-7a73718366c7/volumes"
Nov 04 01:27:34.483302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.483287    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" path="/var/lib/kubelet/pods/9c7f34d0-b8a1-418f-895b-3a5b61c57bda/volumes"
Nov 04 01:27:34.489636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.489615    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"]
Nov 04 01:27:34.497927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:34.497899    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cffa80/simulator-w2-cffa80-kserve-84f9fcff57-x6w7j"]
Nov 04 01:27:35.006919 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:35.005865    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:35.220144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:35.220106    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:35.429009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:35.423528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:35.540009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:35.534806    5797 generic.go:356] "Generic (PLEG): container finished" podID="0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" containerID="d51237da76354521c89e1d51bb3d3de947f0f07043ea017403b429338adc3a58" exitCode=0
Nov 04 01:27:35.540009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:35.535081    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x" event={"ID":"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40","Type":"ContainerDied","Data":"d51237da76354521c89e1d51bb3d3de947f0f07043ea017403b429338adc3a58"}
Nov 04 01:27:35.766950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:35.766853    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:36.008194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.008158    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:36.480964 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.480929    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" path="/var/lib/kubelet/pods/eb767966-ca5f-49cd-80aa-1728469ea3b1/volumes"
Nov 04 01:27:36.787296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.787197    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:36.869811 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.869778    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca\") pod \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") "
Nov 04 01:27:36.869985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.869829    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8\") pod \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") "
Nov 04 01:27:36.869985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.869958    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca\") pod \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\" (UID: \"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40\") "
Nov 04 01:27:36.870362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.870339    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca" pod "0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" (UID: "0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:36.870719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.870702    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:36.872085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.872054    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca" pod "0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" (UID: "0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:36.872189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.872132    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8" pod "0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" (UID: "0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40"). InnerVolumeSpecName "kube-api-access-c5wp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:36.971862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.971820    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:36.971862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:36.971857    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5wp8\" (UniqueName: \"kubernetes.io/projected/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40-kube-api-access-c5wp8\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:37.363080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:37.362975    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:37.553457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:37.553420    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x" event={"ID":"0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40","Type":"ContainerDied","Data":"969e913585b92f31240bbfe297efa39c0b470b13f65e6e5a449f2a2e3ce14e8c"}
Nov 04 01:27:37.553820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:37.553461    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"
Nov 04 01:27:37.553820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:37.553483    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969e913585b92f31240bbfe297efa39c0b470b13f65e6e5a449f2a2e3ce14e8c"
Nov 04 01:27:37.886728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:37.886698    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:38.594770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:38.594738    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:38.799012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:38.797537    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:39.119436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:39.119402    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:39.589026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:39.588898    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:40.459765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:40.459730    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:40.715350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:40.715255    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:41.126078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.126030    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main" containerID="cri-o://82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf" gracePeriod=45
Nov 04 01:27:41.140845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.140804    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" containerID="cri-o://1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8" gracePeriod=45
Nov 04 01:27:41.145447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.145415    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:41.239016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.238961    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused"
Nov 04 01:27:41.479636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.479607    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:27:41.481822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.481798    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:27:41.567105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567070    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm\") pod \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") "
Nov 04 01:27:41.567353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567130    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm\") pod \"2a407479-18d4-4b04-ba29-7961fcf5f538\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") "
Nov 04 01:27:41.567353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567187    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home\") pod \"2a407479-18d4-4b04-ba29-7961fcf5f538\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") "
Nov 04 01:27:41.567353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567222    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt\" (UniqueName: \"kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt\") pod \"2a407479-18d4-4b04-ba29-7961fcf5f538\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") "
Nov 04 01:27:41.567353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567316    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home\") pod \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567362    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs\") pod \"2a407479-18d4-4b04-ba29-7961fcf5f538\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567561    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir\") pod \"2a407479-18d4-4b04-ba29-7961fcf5f538\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567632    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs\") pod \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567701    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache\") pod \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567713    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home" pod "2a407479-18d4-4b04-ba29-7961fcf5f538" (UID: "2a407479-18d4-4b04-ba29-7961fcf5f538"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567805    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v\" (UniqueName: \"kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v\") pod \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567838    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache\") pod \"2a407479-18d4-4b04-ba29-7961fcf5f538\" (UID: \"2a407479-18d4-4b04-ba29-7961fcf5f538\") "
Nov 04 01:27:41.568118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.567880    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir\") pod \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\" (UID: \"d8c17db7-cbad-41ff-92f4-58bc2d4a6458\") "
Nov 04 01:27:41.568795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.568682    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.569168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.568890    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir" pod "2a407479-18d4-4b04-ba29-7961fcf5f538" (UID: "2a407479-18d4-4b04-ba29-7961fcf5f538"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.569168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.568928    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home" pod "d8c17db7-cbad-41ff-92f4-58bc2d4a6458" (UID: "d8c17db7-cbad-41ff-92f4-58bc2d4a6458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.569531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.569507    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache" pod "2a407479-18d4-4b04-ba29-7961fcf5f538" (UID: "2a407479-18d4-4b04-ba29-7961fcf5f538"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.569838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.569814    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir" pod "d8c17db7-cbad-41ff-92f4-58bc2d4a6458" (UID: "d8c17db7-cbad-41ff-92f4-58bc2d4a6458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.569957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.569921    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache" pod "d8c17db7-cbad-41ff-92f4-58bc2d4a6458" (UID: "d8c17db7-cbad-41ff-92f4-58bc2d4a6458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.571447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.571425    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm" pod "d8c17db7-cbad-41ff-92f4-58bc2d4a6458" (UID: "d8c17db7-cbad-41ff-92f4-58bc2d4a6458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.572705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.572647    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs" pod "d8c17db7-cbad-41ff-92f4-58bc2d4a6458" (UID: "d8c17db7-cbad-41ff-92f4-58bc2d4a6458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:41.572705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.572670    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs" pod "2a407479-18d4-4b04-ba29-7961fcf5f538" (UID: "2a407479-18d4-4b04-ba29-7961fcf5f538"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:41.572705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.572695    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v" pod "d8c17db7-cbad-41ff-92f4-58bc2d4a6458" (UID: "d8c17db7-cbad-41ff-92f4-58bc2d4a6458"). InnerVolumeSpecName "kube-api-access-svv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:41.573611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.573581    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm" pod "2a407479-18d4-4b04-ba29-7961fcf5f538" (UID: "2a407479-18d4-4b04-ba29-7961fcf5f538"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:27:41.574331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.574307    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt" pod "2a407479-18d4-4b04-ba29-7961fcf5f538" (UID: "2a407479-18d4-4b04-ba29-7961fcf5f538"). InnerVolumeSpecName "kube-api-access-n45pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:41.601301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.601264    5797 generic.go:356] "Generic (PLEG): container finished" podID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerID="82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf" exitCode=0
Nov 04 01:27:41.602117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.601310    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" event={"ID":"d8c17db7-cbad-41ff-92f4-58bc2d4a6458","Type":"ContainerDied","Data":"82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf"}
Nov 04 01:27:41.602117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.601366    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2" event={"ID":"d8c17db7-cbad-41ff-92f4-58bc2d4a6458","Type":"ContainerDied","Data":"69c5f9ac3d68b2127758d2cdb8334776ba5e49b4620afc51aedb91e3330459f5"}
Nov 04 01:27:41.602117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.601321    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"
Nov 04 01:27:41.602117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.601390    5797 scope.go:122] "RemoveContainer" containerID="82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf"
Nov 04 01:27:41.603774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.603750    5797 generic.go:356] "Generic (PLEG): container finished" podID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerID="1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8" exitCode=0
Nov 04 01:27:41.603961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.603827    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" event={"ID":"2a407479-18d4-4b04-ba29-7961fcf5f538","Type":"ContainerDied","Data":"1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8"}
Nov 04 01:27:41.603961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.603859    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv" event={"ID":"2a407479-18d4-4b04-ba29-7961fcf5f538","Type":"ContainerDied","Data":"69669c641580e8bfcf882c5df45b38a8922ca646e5f1a266140533ad246f4630"}
Nov 04 01:27:41.603961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.603889    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"
Nov 04 01:27:41.613382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.613358    5797 scope.go:122] "RemoveContainer" containerID="82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf"
Nov 04 01:27:41.613675 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:41.613657    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf\": container with ID starting with 82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf not found: ID does not exist" containerID="82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf"
Nov 04 01:27:41.613759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.613684    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf"} err="failed to get container status \"82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf\": rpc error: code = NotFound desc = could not find container \"82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf\": container with ID starting with 82bde43fa0a3743baeec0868d27eb22a36fc1b6bfa4059de5940d736b40f7baf not found: ID does not exist"
Nov 04 01:27:41.613759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.613699    5797 scope.go:122] "RemoveContainer" containerID="1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8"
Nov 04 01:27:41.624391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.624374    5797 scope.go:122] "RemoveContainer" containerID="1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8"
Nov 04 01:27:41.624698 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:41.624676    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8\": container with ID starting with 1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8 not found: ID does not exist" containerID="1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8"
Nov 04 01:27:41.624801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.624705    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8"} err="failed to get container status \"1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8\": rpc error: code = NotFound desc = could not find container \"1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8\": container with ID starting with 1f82dccd56c8073c17ed1eaaf19514bfe8133e26fcc2d7f20459645c52c344a8 not found: ID does not exist"
Nov 04 01:27:41.634004 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.633968    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"]
Nov 04 01:27:41.637443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.637419    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-875528/simulator-w5-875528-kserve-7bbfcbdb95-cj9tv"]
Nov 04 01:27:41.658426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.658395    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"]
Nov 04 01:27:41.667399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.667310    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-875528/distinct-w5-875528-kserve-77c8c8df7-brdl2"]
Nov 04 01:27:41.670704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670675    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670702    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670715    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670724    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670735    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n45pt\" (UniqueName: \"kubernetes.io/projected/2a407479-18d4-4b04-ba29-7961fcf5f538-kube-api-access-n45pt\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670743    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670751    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a407479-18d4-4b04-ba29-7961fcf5f538-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670759    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a407479-18d4-4b04-ba29-7961fcf5f538-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670768    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670775    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.670898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.670784    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svv6v\" (UniqueName: \"kubernetes.io/projected/d8c17db7-cbad-41ff-92f4-58bc2d4a6458-kube-api-access-svv6v\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:41.908961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:41.908920    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:42.480446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:42.480362    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" path="/var/lib/kubelet/pods/2a407479-18d4-4b04-ba29-7961fcf5f538/volumes"
Nov 04 01:27:42.481052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:42.481035    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" path="/var/lib/kubelet/pods/d8c17db7-cbad-41ff-92f4-58bc2d4a6458/volumes"
Nov 04 01:27:42.565163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:42.565125    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:43.033263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.033217    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"]
Nov 04 01:27:43.036450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.036421    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e7c225-b796p"]
Nov 04 01:27:43.084529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.084498    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"]
Nov 04 01:27:43.085182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085163    5797 cpu_manager.go:414] "containerMap: removing container" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy"
Nov 04 01:27:43.085182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085181    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085197    5797 cpu_manager.go:414] "containerMap: removing container" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085202    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085216    5797 cpu_manager.go:414] "containerMap: removing container" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085221    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085228    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" containerName="revoke-keys"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085233    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" containerName="revoke-keys"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085248    5797 cpu_manager.go:414] "containerMap: removing container" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085253    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085271    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085276    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085284    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main"
Nov 04 01:27:43.085327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085290    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085380    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb767966-ca5f-49cd-80aa-1728469ea3b1" containerName="main"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085395    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a407479-18d4-4b04-ba29-7961fcf5f538" containerName="main"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085404    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9ad66a-bede-4e9a-94ea-f9dd9a924f61" containerName="istio-proxy"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085411    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bc8ba2-e063-4d0e-b12d-7a73718366c7" containerName="istio-proxy"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085417    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7f34d0-b8a1-418f-895b-3a5b61c57bda" containerName="istio-proxy"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085423    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c17db7-cbad-41ff-92f4-58bc2d4a6458" containerName="main"
Nov 04 01:27:43.085759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.085429    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" containerName="revoke-keys"
Nov 04 01:27:43.090404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.090385    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.094579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.094545    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:27:43.094789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.094770    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:27:43.098689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.098665    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"]
Nov 04 01:27:43.193613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.193578    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.193819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.193641    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtvjw\" (UniqueName: \"kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.193901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.193820    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.289443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.289352    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:43.294801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.294774    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.294961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.294948    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.295048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.295010    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtvjw\" (UniqueName: \"kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.295589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.295565    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.297182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.297160    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.307631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.307607    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtvjw\" (UniqueName: \"kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw\") pod \"maas-api-revoke-sub-979dd2f4-rmpkg\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.412200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.412171    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:43.580320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.580231    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"]
Nov 04 01:27:43.700051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.699961    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"]
Nov 04 01:27:43.766787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.766753    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:43.999141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:43.996840    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:44.242205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:44.242173    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:44.502194 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:44.502159    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec4f014-0a81-40fb-82f9-6b1c291b4f0c" path="/var/lib/kubelet/pods/8ec4f014-0a81-40fb-82f9-6b1c291b4f0c/volumes"
Nov 04 01:27:44.538032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:44.537973    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:44.638214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:44.638182    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg" event={"ID":"5738dfe1-c350-42e8-8a60-82e40169ec3c","Type":"ContainerStarted","Data":"79a7259bb11df76f6032504ac490e6a15d4718cedee2e6f66098647c4e2c1704"}
Nov 04 01:27:44.638618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:44.638242    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg" event={"ID":"5738dfe1-c350-42e8-8a60-82e40169ec3c","Type":"ContainerStarted","Data":"99b864a4df274d50976371c64756dee0eb9d0e452a5e9650177f61ca89289277"}
Nov 04 01:27:44.660149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:44.660097    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg" podStartSLOduration=1.660083448 podStartE2EDuration="1.660083448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:44.656279808 +0000 UTC m=+1452.298186059" watchObservedRunningTime="2026-11-04 01:27:44.660083448 +0000 UTC m=+1452.301989695"
Nov 04 01:27:45.299165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:45.299114    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:45.497644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:45.497591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:45.646548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:45.646515    5797 generic.go:356] "Generic (PLEG): container finished" podID="5738dfe1-c350-42e8-8a60-82e40169ec3c" containerID="79a7259bb11df76f6032504ac490e6a15d4718cedee2e6f66098647c4e2c1704" exitCode=0
Nov 04 01:27:45.646952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:45.646592    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg" event={"ID":"5738dfe1-c350-42e8-8a60-82e40169ec3c","Type":"ContainerDied","Data":"79a7259bb11df76f6032504ac490e6a15d4718cedee2e6f66098647c4e2c1704"}
Nov 04 01:27:46.188438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.188400    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:46.388764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.388728    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:46.881967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.881941    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:46.999062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.989856    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca\") pod \"5738dfe1-c350-42e8-8a60-82e40169ec3c\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") "
Nov 04 01:27:46.999062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.989931    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca\") pod \"5738dfe1-c350-42e8-8a60-82e40169ec3c\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") "
Nov 04 01:27:46.999062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.990124    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw\" (UniqueName: \"kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw\") pod \"5738dfe1-c350-42e8-8a60-82e40169ec3c\" (UID: \"5738dfe1-c350-42e8-8a60-82e40169ec3c\") "
Nov 04 01:27:46.999062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:46.990473    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:47.004754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.004716    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw" pod "5738dfe1-c350-42e8-8a60-82e40169ec3c" (UID: "5738dfe1-c350-42e8-8a60-82e40169ec3c"). InnerVolumeSpecName "kube-api-access-dtvjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:47.008281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.007463    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca" pod "5738dfe1-c350-42e8-8a60-82e40169ec3c" (UID: "5738dfe1-c350-42e8-8a60-82e40169ec3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:27:47.024008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.021597    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca" pod "5738dfe1-c350-42e8-8a60-82e40169ec3c" (UID: "5738dfe1-c350-42e8-8a60-82e40169ec3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:47.140752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.139417    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtvjw\" (UniqueName: \"kubernetes.io/projected/5738dfe1-c350-42e8-8a60-82e40169ec3c-kube-api-access-dtvjw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:47.140752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.139454    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:47.140752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.139470    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5738dfe1-c350-42e8-8a60-82e40169ec3c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:47.204908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.204873    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:47.409981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.409941    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:47.689816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.689093    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg" event={"ID":"5738dfe1-c350-42e8-8a60-82e40169ec3c","Type":"ContainerDied","Data":"99b864a4df274d50976371c64756dee0eb9d0e452a5e9650177f61ca89289277"}
Nov 04 01:27:47.689816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.689154    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b864a4df274d50976371c64756dee0eb9d0e452a5e9650177f61ca89289277"
Nov 04 01:27:47.689816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:47.689224    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"
Nov 04 01:27:48.022073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:48.017304    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:48.581626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:48.581058    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing" containerID="cri-o://5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f" gracePeriod=25
Nov 04 01:27:48.989010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:48.986622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:49.075327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.075297    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:27:49.221725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.221683    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume\") pod \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") "
Nov 04 01:27:49.221902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.221801    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj\" (UniqueName: \"kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj\") pod \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\" (UID: \"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8\") "
Nov 04 01:27:49.222173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.222146    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume" pod "4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" (UID: "4fb22100-7a0e-49ee-9ab4-66b2a54c34c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:27:49.222425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.222406    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:49.224008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.223969    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj" pod "4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" (UID: "4fb22100-7a0e-49ee-9ab4-66b2a54c34c8"). InnerVolumeSpecName "kube-api-access-dwbvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:27:49.324406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.324371    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbvj\" (UniqueName: \"kubernetes.io/projected/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8-kube-api-access-dwbvj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:27:49.647104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.647011    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:49.765054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.758574    5797 generic.go:356] "Generic (PLEG): container finished" podID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerID="5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f" exitCode=0
Nov 04 01:27:49.765054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.758613    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" event={"ID":"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8","Type":"ContainerDied","Data":"5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f"}
Nov 04 01:27:49.765054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.758692    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm" event={"ID":"4fb22100-7a0e-49ee-9ab4-66b2a54c34c8","Type":"ContainerDied","Data":"705ebf38bfe0ce78a8cc08b8d3f69c9e2f4cfcabd1ae3fc74d0b13b8fda514c6"}
Nov 04 01:27:49.765054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.758719    5797 scope.go:122] "RemoveContainer" containerID="5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f"
Nov 04 01:27:49.765054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.758952    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"
Nov 04 01:27:49.774394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.774368    5797 scope.go:122] "RemoveContainer" containerID="5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f"
Nov 04 01:27:49.774701 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:49.774681    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f\": container with ID starting with 5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f not found: ID does not exist" containerID="5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f"
Nov 04 01:27:49.774767 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.774713    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f"} err="failed to get container status \"5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f\": rpc error: code = NotFound desc = could not find container \"5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f\": container with ID starting with 5f14281b1ec166431b3f033f593613dbd792ef3e3b0be8e0230555e6d895119f not found: ID does not exist"
Nov 04 01:27:49.784745 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.784714    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"]
Nov 04 01:27:49.789258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.789226    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-875528-7594cb59f5-v5jsm"]
Nov 04 01:27:49.843822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:49.843769    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:50.053585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:50.053496    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:50.266690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:50.266655    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:50.479665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:50.479632    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" path="/var/lib/kubelet/pods/4fb22100-7a0e-49ee-9ab4-66b2a54c34c8/volumes"
Nov 04 01:27:50.595340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:50.595303    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:50.966961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:50.966925    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:51.044230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.044196    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"]
Nov 04 01:27:51.044945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.044924    5797 cpu_manager.go:414] "containerMap: removing container" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing"
Nov 04 01:27:51.045016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.044947    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing"
Nov 04 01:27:51.045016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.044983    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5738dfe1-c350-42e8-8a60-82e40169ec3c" containerName="revoke-keys"
Nov 04 01:27:51.045016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.045005    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5738dfe1-c350-42e8-8a60-82e40169ec3c" containerName="revoke-keys"
Nov 04 01:27:51.045123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.045100    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb22100-7a0e-49ee-9ab4-66b2a54c34c8" containerName="payload-pre-processing"
Nov 04 01:27:51.045123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.045115    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5738dfe1-c350-42e8-8a60-82e40169ec3c" containerName="revoke-keys"
Nov 04 01:27:51.050752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.050724    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.061383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.057042    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6ecd1f-openshift-default-dockercfg-dzphl\""
Nov 04 01:27:51.062249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.062224    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"]
Nov 04 01:27:51.152662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.152624    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.152830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.152668    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.152830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.152779    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bv9\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.152904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.152837    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.152961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.152941    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.153097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.153082    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.153176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.153163    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.153210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.153192    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.153249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.153223    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.179567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.179540    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:51.270147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.270049    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.272532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.272498    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.272769 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.272749    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.272897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.272882    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.273063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.273046    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.273193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.273178    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.273344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.273328    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bv9\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.273492 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.273478    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.273671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.273657    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.284070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.275594    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.285062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.277206    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.301332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.301294    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.303628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.303352    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.303628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.303587    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.304195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.304174    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.310625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.310570    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.310625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.310592    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bv9\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.312900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.312834    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token\") pod \"e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") " pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.370722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.370344    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:51.382116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.380032    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:51.585046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.584940    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:51.708136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.708104    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"]
Nov 04 01:27:51.708738 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:27:51.708711    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc680caa3_cc12_4f0a_b909_cca4a3e9adcb.slice/crio-930b1b92f4e64e49c70f2603df5330600204834f65af1936d6093bf664a69c59 WatchSource:0}: Error finding container 930b1b92f4e64e49c70f2603df5330600204834f65af1936d6093bf664a69c59: Status 404 returned error can't find the container with id 930b1b92f4e64e49c70f2603df5330600204834f65af1936d6093bf664a69c59
Nov 04 01:27:51.711404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.711369    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:27:51.711485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.711431    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:27:51.711485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.711460    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:27:51.779045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:51.778968    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" event={"ID":"c680caa3-cc12-4f0a-b909-cca4a3e9adcb","Type":"ContainerStarted","Data":"930b1b92f4e64e49c70f2603df5330600204834f65af1936d6093bf664a69c59"}
Nov 04 01:27:52.042009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:52.033754    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:52.759013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:52.753583    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:52.812010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:52.811437    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" event={"ID":"c680caa3-cc12-4f0a-b909-cca4a3e9adcb","Type":"ContainerStarted","Data":"f98d4d2b4f47c74e9fbb279d985f5a2a1a519d50227ade3640e8f5a83324a2b5"}
Nov 04 01:27:52.963012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:52.962768    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:53.290639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.289453    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:53.386029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.385107    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:53.429119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.421128    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:53.472012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.465978    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podStartSLOduration=2.465951509 podStartE2EDuration="2.465951509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:52.909265591 +0000 UTC m=+1460.551171864" watchObservedRunningTime="2026-11-04 01:27:53.465951509 +0000 UTC m=+1461.107857760"
Nov 04 01:27:53.730153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.730109    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:53.828015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.825567    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:53.838032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:53.831241    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:27:54.210066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:54.210029    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:54.622569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:54.622533    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:55.533021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:55.532927    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:55.745192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:55.745158    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:55.986643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:55.986604    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:56.257277 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:27:56.257189    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:27:57.090178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:57.090144    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:57.334086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:57.334034    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:57.585168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:57.571193    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:57.810502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:57.810462    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:58.020783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.020747    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:58.145334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.145304    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"]
Nov 04 01:27:58.147062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.147026    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy" containerID="cri-o://5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48" gracePeriod=30
Nov 04 01:27:58.201497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.201459    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"]
Nov 04 01:27:58.207118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.207087    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"]
Nov 04 01:27:58.207530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.207499    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy" containerID="cri-o://e13db9797f0c4562db17be0664c1ff7ff345e38a4b91ab10c42f12c40c68f2ea" gracePeriod=30
Nov 04 01:27:58.208055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.207950    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy" containerID="cri-o://6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910" gracePeriod=30
Nov 04 01:27:58.242464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.237681    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:58.453014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.448572    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:58.670530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:58.670495    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:59.579520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:59.577840    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:27:59.860962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:27:59.860880    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:00.646122 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:00.646069    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:00.858535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:00.858500    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:01.637617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:01.637540    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:01.864556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:01.864526    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:02.282884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:02.282851    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:02.510573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:02.510535    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:03.027341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.027307    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"]
Nov 04 01:28:03.040023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.039982    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ac610-7tzpr"]
Nov 04 01:28:03.410014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.405955    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:03.573510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.573448    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:28:03.671423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671390    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671483    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671635    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671685    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671698    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.671757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671732    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671768    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671884    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.671926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.671921    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.672295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672130    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.672295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672171    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs\") pod \"3fa7e470-2922-43cd-a478-714d01ed9e64\" (UID: \"3fa7e470-2922-43cd-a478-714d01ed9e64\") "
Nov 04 01:28:03.672295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672231    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.672523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672325    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:03.672896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672873    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.673022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672902    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa7e470-2922-43cd-a478-714d01ed9e64-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.673022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672917    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.673022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.672930    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.673233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.673144    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.674040 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.674017    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:03.674367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.674347    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.674446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.674365    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:03.674446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.674393    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8" pod "3fa7e470-2922-43cd-a478-714d01ed9e64" (UID: "3fa7e470-2922-43cd-a478-714d01ed9e64"). InnerVolumeSpecName "kube-api-access-g54s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:03.774371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.774332    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.774545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.774374    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54s8\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-kube-api-access-g54s8\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.774545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.774393    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.774545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.774407    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa7e470-2922-43cd-a478-714d01ed9e64-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.774545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.774421    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa7e470-2922-43cd-a478-714d01ed9e64-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:03.890115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.890081    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:28:03.930665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.930627    5797 generic.go:356] "Generic (PLEG): container finished" podID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerID="e13db9797f0c4562db17be0664c1ff7ff345e38a4b91ab10c42f12c40c68f2ea" exitCode=0
Nov 04 01:28:03.930665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.930651    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" event={"ID":"e43f7199-d45f-4e71-b5dc-7bd24a975284","Type":"ContainerDied","Data":"e13db9797f0c4562db17be0664c1ff7ff345e38a4b91ab10c42f12c40c68f2ea"}
Nov 04 01:28:03.932505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.932481    5797 generic.go:356] "Generic (PLEG): container finished" podID="98ed529b-8904-4f22-baae-e7635a3090b3" containerID="6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910" exitCode=0
Nov 04 01:28:03.932656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.932555    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"
Nov 04 01:28:03.932656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.932563    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" event={"ID":"98ed529b-8904-4f22-baae-e7635a3090b3","Type":"ContainerDied","Data":"6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910"}
Nov 04 01:28:03.932656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.932621    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj" event={"ID":"98ed529b-8904-4f22-baae-e7635a3090b3","Type":"ContainerDied","Data":"bc0dcbdd31db8a5bc98769bfa4d3a884054927e995e48fba68371cc0f5ad148e"}
Nov 04 01:28:03.932656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.932650    5797 scope.go:122] "RemoveContainer" containerID="6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910"
Nov 04 01:28:03.934751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.934724    5797 generic.go:356] "Generic (PLEG): container finished" podID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerID="5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48" exitCode=0
Nov 04 01:28:03.934840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.934760    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" event={"ID":"3fa7e470-2922-43cd-a478-714d01ed9e64","Type":"ContainerDied","Data":"5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48"}
Nov 04 01:28:03.934840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.934774    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"
Nov 04 01:28:03.934840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.934801    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd" event={"ID":"3fa7e470-2922-43cd-a478-714d01ed9e64","Type":"ContainerDied","Data":"52f97cd2521070597d272378c8982088b7ddfa0f70c7bb6c88e74d2526c47900"}
Nov 04 01:28:03.957054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.957023    5797 scope.go:122] "RemoveContainer" containerID="6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910"
Nov 04 01:28:03.957459 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:03.957427    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910\": container with ID starting with 6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910 not found: ID does not exist" containerID="6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910"
Nov 04 01:28:03.957564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.957472    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910"} err="failed to get container status \"6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910\": rpc error: code = NotFound desc = could not find container \"6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910\": container with ID starting with 6e823f020dc7b44b5fc8f6a447ada7a2b800e9e9db2009f31661f566e5f98910 not found: ID does not exist"
Nov 04 01:28:03.957564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.957495    5797 scope.go:122] "RemoveContainer" containerID="5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48"
Nov 04 01:28:03.961487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.961393    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:28:03.969809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.969788    5797 scope.go:122] "RemoveContainer" containerID="5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48"
Nov 04 01:28:03.970154 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:03.970123    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48\": container with ID starting with 5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48 not found: ID does not exist" containerID="5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48"
Nov 04 01:28:03.970388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.970163    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48"} err="failed to get container status \"5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48\": rpc error: code = NotFound desc = could not find container \"5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48\": container with ID starting with 5a0cbfc6355be57cfdacb9e2a17b52e3993ee51c034d8ea3a4e5346340b78a48 not found: ID does not exist"
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.976976    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.977496    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.977651    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.977712    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.977961    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.978013    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.978040    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.978090    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.978148    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data\") pod \"98ed529b-8904-4f22-baae-e7635a3090b3\" (UID: \"98ed529b-8904-4f22-baae-e7635a3090b3\") "
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.979586    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.979937    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:03.981031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.980521    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"]
Nov 04 01:28:03.984515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.984216    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:03.984962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.984739    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.985310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.985122    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.985610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.985430    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.986012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.985728    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:03.986441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.986412    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:03.987046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.987014    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-ct4pd"]
Nov 04 01:28:03.990538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:03.990519    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t" pod "98ed529b-8904-4f22-baae-e7635a3090b3" (UID: "98ed529b-8904-4f22-baae-e7635a3090b3"). InnerVolumeSpecName "kube-api-access-t4h2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085441    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085518    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085555    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085613    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085660    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085748    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085843    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.085903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085904    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.086808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.085950    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token\") pod \"e43f7199-d45f-4e71-b5dc-7bd24a975284\" (UID: \"e43f7199-d45f-4e71-b5dc-7bd24a975284\") "
Nov 04 01:28:04.086808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086751    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ed529b-8904-4f22-baae-e7635a3090b3-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.086808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086778    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.086808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086793    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.086808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086809    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4h2t\" (UniqueName: \"kubernetes.io/projected/98ed529b-8904-4f22-baae-e7635a3090b3-kube-api-access-t4h2t\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.087108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086824    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ed529b-8904-4f22-baae-e7635a3090b3-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.087108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086838    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.087108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086854    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.087108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086868    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.087108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.086880    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ed529b-8904-4f22-baae-e7635a3090b3-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.144275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.105262    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:04.144275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.105581    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:04.144275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.105845    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:04.144275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.122682    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:04.144275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.142399    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:04.197594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.197257    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.197594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.197297    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.197594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.197313    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.197594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.197327    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43f7199-d45f-4e71-b5dc-7bd24a975284-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.197594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.197343    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.198006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.197645    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:04.211026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.204066    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:04.238029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.225164    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:04.238029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.225299    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p" pod "e43f7199-d45f-4e71-b5dc-7bd24a975284" (UID: "e43f7199-d45f-4e71-b5dc-7bd24a975284"). InnerVolumeSpecName "kube-api-access-xsd6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:04.291166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.290751    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:04.298686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.298590    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsd6p\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-kube-api-access-xsd6p\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.298686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.298625    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.298686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.298641    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.298686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.298658    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43f7199-d45f-4e71-b5dc-7bd24a975284-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:04.335072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.334813    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"]
Nov 04 01:28:04.341802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.341769    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-h6lgj"]
Nov 04 01:28:04.501065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.498736    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" path="/var/lib/kubelet/pods/3fa7e470-2922-43cd-a478-714d01ed9e64/volumes"
Nov 04 01:28:04.501065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.499722    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a659a7b-4ca9-499e-8059-d6916330925a" path="/var/lib/kubelet/pods/8a659a7b-4ca9-499e-8059-d6916330925a/volumes"
Nov 04 01:28:04.501065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.500455    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" path="/var/lib/kubelet/pods/98ed529b-8904-4f22-baae-e7635a3090b3/volumes"
Nov 04 01:28:04.549072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.547418    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:04.945539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.945507    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"
Nov 04 01:28:04.945762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.945516    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk" event={"ID":"e43f7199-d45f-4e71-b5dc-7bd24a975284","Type":"ContainerDied","Data":"f70edd4a820b0f7052d252b6d91c622f5e8d545218991f9409ea4207a43aa5b0"}
Nov 04 01:28:04.945762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.945643    5797 scope.go:122] "RemoveContainer" containerID="e13db9797f0c4562db17be0664c1ff7ff345e38a4b91ab10c42f12c40c68f2ea"
Nov 04 01:28:04.967313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.967284    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"]
Nov 04 01:28:04.972963 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:04.972927    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cffa80-openshift-default-69c4c7fd8f-4w8wk"]
Nov 04 01:28:05.478952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:05.478886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:05.681449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:05.681418    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:05.883294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:05.883243    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:06.113625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:06.113585    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:06.582061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:06.572694    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:06.591602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:06.591568    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" path="/var/lib/kubelet/pods/e43f7199-d45f-4e71-b5dc-7bd24a975284/volumes"
Nov 04 01:28:06.775761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:06.775729    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:06.977057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:06.977027    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:07.568055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:07.564758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:08.397142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:08.397109    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:08.846096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:08.845278    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:09.284010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:09.283086    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:09.485462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:09.485427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:09.691861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:09.691830    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:09.893563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:09.893530    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:10.128029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:10.127973    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"]
Nov 04 01:28:10.128279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:10.128257    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy" containerID="cri-o://223684cc34028beb4d97c39d2caf26e1029ae0456a40a7c9dd418a4ccc6af764" gracePeriod=30
Nov 04 01:28:10.234041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:10.233487    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:10.640293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:10.640250    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:10.857724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:10.857689    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:11.084008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.083919    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"]
Nov 04 01:28:11.084558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084542    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy"
Nov 04 01:28:11.084605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084560    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy"
Nov 04 01:28:11.084647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084606    5797 cpu_manager.go:414] "containerMap: removing container" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy"
Nov 04 01:28:11.084647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084612    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy"
Nov 04 01:28:11.084647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084624    5797 cpu_manager.go:414] "containerMap: removing container" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy"
Nov 04 01:28:11.084647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084632    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy"
Nov 04 01:28:11.084775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084714    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43f7199-d45f-4e71-b5dc-7bd24a975284" containerName="istio-proxy"
Nov 04 01:28:11.084775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084731    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa7e470-2922-43cd-a478-714d01ed9e64" containerName="istio-proxy"
Nov 04 01:28:11.084775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.084736    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ed529b-8904-4f22-baae-e7635a3090b3" containerName="istio-proxy"
Nov 04 01:28:11.089592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.089572    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.096488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.096461    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:28:11.097617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.097591    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:28:11.103818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.103795    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"]
Nov 04 01:28:11.170840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.170804    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.171047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.170848    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.171184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.171163    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vgz\" (UniqueName: \"kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.272876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.272843    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vgz\" (UniqueName: \"kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.273108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.272903    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.273108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.272939    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.273853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.273817    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.275654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.275631    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.285482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.285448    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vgz\" (UniqueName: \"kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz\") pod \"maas-api-revoke-sub-24c58e0a-zjmdk\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.410655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.409651    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:11.753921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.753886    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"]
Nov 04 01:28:11.754942 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:28:11.754912    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f384bd_68ce_4fd5_8da5_8b5197924582.slice/crio-8f0ff60af9850fdf4ee835b3076a16a7dc0382027b7683495d4be467b0214a28 WatchSource:0}: Error finding container 8f0ff60af9850fdf4ee835b3076a16a7dc0382027b7683495d4be467b0214a28: Status 404 returned error can't find the container with id 8f0ff60af9850fdf4ee835b3076a16a7dc0382027b7683495d4be467b0214a28
Nov 04 01:28:11.890069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:11.889648    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:12.050207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.050115    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk" event={"ID":"70f384bd-68ce-4fd5-8da5-8b5197924582","Type":"ContainerStarted","Data":"8f0ff60af9850fdf4ee835b3076a16a7dc0382027b7683495d4be467b0214a28"}
Nov 04 01:28:12.116214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.116173    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:12.543237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.543203    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"]
Nov 04 01:28:12.547752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.547729    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"]
Nov 04 01:28:12.550205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.550182    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.563281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.563229    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"]
Nov 04 01:28:12.565032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.564971    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.571183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.571162    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"]
Nov 04 01:28:12.571491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.571475    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"]
Nov 04 01:28:12.571712 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.571698    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"]
Nov 04 01:28:12.571851 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.571415    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.642927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.642897    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.643141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.642948    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.643141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.642971    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.643141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643101    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.643263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643160    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.643306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643289    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.643347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643337    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.643401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643387    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.643494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643480    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.643527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643511    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.643603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643587    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.643639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643626    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lr9\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.643678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643653    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.643725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643682    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.643792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643778    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtvf4\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.643849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643808    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.643849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643829    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.643952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643915    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.644042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643954    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.644042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.643969    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.644042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644009    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.644211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644048    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.644211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644078    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22q5f\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.644211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644133    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.644211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644165    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.644211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644193    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.644486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.644311    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746100 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746064    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746111    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746169    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746189    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lr9\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746212    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746246    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746421    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtvf4\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746473    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746506    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.746581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746559    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746590    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746614    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746648    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746685    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746689    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746716    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22q5f\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746756    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.746800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746789    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746817    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746851    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746893    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746934    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.746975    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747045    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747081    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747106    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747131    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747173    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747218    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.747286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747284    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747297    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747319    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747415    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747647    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747683    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747699    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.747939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.747905    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.748350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.748083    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.748350 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.748228    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.748483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.748458    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.748824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.748797    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.749404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.749382    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.750502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.750473    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.751425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.750756    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.751425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.751370    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.752056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.752012    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.752882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.752670    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.755148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.755103    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.764831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.764774    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.764831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.764787    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.765025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.764930    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lr9\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.765435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.765408    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.765576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.765551    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22q5f\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.765936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.765919    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtvf4\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4\") pod \"e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") " pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:12.880646 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.880608    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:12.901412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.901378    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:12.904530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:12.904498    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:13.051637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.048852    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:13.105588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.104890    5797 generic.go:356] "Generic (PLEG): container finished" podID="70f384bd-68ce-4fd5-8da5-8b5197924582" containerID="93665e8d732092393eebe33061f2aff1eb4d90c2f881a38e4296b3b2cf9c2e1a" exitCode=0
Nov 04 01:28:13.105588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.105138    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk" event={"ID":"70f384bd-68ce-4fd5-8da5-8b5197924582","Type":"ContainerDied","Data":"93665e8d732092393eebe33061f2aff1eb4d90c2f881a38e4296b3b2cf9c2e1a"}
Nov 04 01:28:13.282708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.282669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:13.291155 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:28:13.291128    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cded768_46fe_41cb_a166_7b435f6836b3.slice/crio-534557b58b5beafcd94182d9e7542c6348431e3ad3b2c88abdc540e84b90da3b WatchSource:0}: Error finding container 534557b58b5beafcd94182d9e7542c6348431e3ad3b2c88abdc540e84b90da3b: Status 404 returned error can't find the container with id 534557b58b5beafcd94182d9e7542c6348431e3ad3b2c88abdc540e84b90da3b
Nov 04 01:28:13.291300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.291273    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"]
Nov 04 01:28:13.294211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.294175    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.294336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.294253    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.294336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.294308    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.446375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.446341    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"]
Nov 04 01:28:13.504536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.504492    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.522395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.522345    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.522683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.522642    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.624156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.624115    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.624255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.624223    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.624321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.624282    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:13.648186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:13.644474    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"]
Nov 04 01:28:14.122181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.121212    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" event={"ID":"8cded768-46fe-41cb-a166-7b435f6836b3","Type":"ContainerStarted","Data":"219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5"}
Nov 04 01:28:14.122181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.121280    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" event={"ID":"8cded768-46fe-41cb-a166-7b435f6836b3","Type":"ContainerStarted","Data":"534557b58b5beafcd94182d9e7542c6348431e3ad3b2c88abdc540e84b90da3b"}
Nov 04 01:28:14.128364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.128330    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" event={"ID":"0ba89adb-97d7-404f-958c-d06cd49a8b68","Type":"ContainerStarted","Data":"60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973"}
Nov 04 01:28:14.128661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.128623    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" event={"ID":"0ba89adb-97d7-404f-958c-d06cd49a8b68","Type":"ContainerStarted","Data":"38bdb5d83b37ae4e4385f7ebf5714f98ce0c6ae12e6a2a21973d35dad49cde37"}
Nov 04 01:28:14.132429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.132406    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" event={"ID":"92bb280f-0dc4-4e9e-b443-1349080c5a2e","Type":"ContainerStarted","Data":"206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5"}
Nov 04 01:28:14.132617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.132602    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" event={"ID":"92bb280f-0dc4-4e9e-b443-1349080c5a2e","Type":"ContainerStarted","Data":"a21a9e9c82fba9e03663217d8d92d291c6cd17b907fafb43032f75a78369951e"}
Nov 04 01:28:14.207670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.207600    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podStartSLOduration=2.207581099 podStartE2EDuration="2.207581099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:14.206645734 +0000 UTC m=+1481.848552010" watchObservedRunningTime="2026-11-04 01:28:14.207581099 +0000 UTC m=+1481.849487370"
Nov 04 01:28:14.265015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.251746    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podStartSLOduration=2.251723577 podStartE2EDuration="2.251723577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:14.171910742 +0000 UTC m=+1481.813817021" watchObservedRunningTime="2026-11-04 01:28:14.251723577 +0000 UTC m=+1481.893629828"
Nov 04 01:28:14.281022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:14.276688    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podStartSLOduration=2.2766646440000002 podStartE2EDuration="2.276664644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:14.245343738 +0000 UTC m=+1481.887249989" watchObservedRunningTime="2026-11-04 01:28:14.276664644 +0000 UTC m=+1481.918570896"
Nov 04 01:28:15.055962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.054731    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 04 01:28:15.055962 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.054795    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 04 01:28:15.140014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.123160    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 04 01:28:15.140014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.123262    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 04 01:28:15.140014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.132161    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 04 01:28:15.140014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.132268    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 04 01:28:15.236611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.236570    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:15.351605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.246728    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:15.351893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.351870    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:15.352081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.352067    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:15.512164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.489349    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:15.519033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.518977    5797 generic.go:356] "Generic (PLEG): container finished" podID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerID="223684cc34028beb4d97c39d2caf26e1029ae0456a40a7c9dd418a4ccc6af764" exitCode=0
Nov 04 01:28:15.544044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.529087    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" event={"ID":"94d933d6-8278-432e-bdf4-7bbd5d63f8b7","Type":"ContainerDied","Data":"223684cc34028beb4d97c39d2caf26e1029ae0456a40a7c9dd418a4ccc6af764"}
Nov 04 01:28:15.544044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.529207    5797 scope.go:122] "RemoveContainer" containerID="8ee2c20f41f18de169acce7a515f4bf24c8d59cf0d340f9b941c11f38fa45e21"
Nov 04 01:28:15.646520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.645284    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca\") pod \"70f384bd-68ce-4fd5-8da5-8b5197924582\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") "
Nov 04 01:28:15.646520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.645378    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz\" (UniqueName: \"kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz\") pod \"70f384bd-68ce-4fd5-8da5-8b5197924582\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") "
Nov 04 01:28:15.646520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.645522    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca\") pod \"70f384bd-68ce-4fd5-8da5-8b5197924582\" (UID: \"70f384bd-68ce-4fd5-8da5-8b5197924582\") "
Nov 04 01:28:15.666520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.666480    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca" pod "70f384bd-68ce-4fd5-8da5-8b5197924582" (UID: "70f384bd-68ce-4fd5-8da5-8b5197924582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:15.749222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.749188    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:15.778241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.778205    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz" pod "70f384bd-68ce-4fd5-8da5-8b5197924582" (UID: "70f384bd-68ce-4fd5-8da5-8b5197924582"). InnerVolumeSpecName "kube-api-access-74vgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:15.843316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.838432    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca" pod "70f384bd-68ce-4fd5-8da5-8b5197924582" (UID: "70f384bd-68ce-4fd5-8da5-8b5197924582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:28:15.886021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.883450    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vgz\" (UniqueName: \"kubernetes.io/projected/70f384bd-68ce-4fd5-8da5-8b5197924582-kube-api-access-74vgz\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:15.886021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.883493    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f384bd-68ce-4fd5-8da5-8b5197924582-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:15.931316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.931221    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 04 01:28:15.944404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.931567    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 04 01:28:15.989012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:15.947517    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 04 01:28:16.100024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.099351    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 04 01:28:16.141056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.131168    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:16.141056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.131332    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 04 01:28:16.141056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.131387    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 04 01:28:16.335654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.335571    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:16.643836 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.643797    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 04 01:28:16.680789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.680750    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:16.698577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.698542    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"
Nov 04 01:28:16.720201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.699731    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk" event={"ID":"70f384bd-68ce-4fd5-8da5-8b5197924582","Type":"ContainerDied","Data":"8f0ff60af9850fdf4ee835b3076a16a7dc0382027b7683495d4be467b0214a28"}
Nov 04 01:28:16.720539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.720496    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0ff60af9850fdf4ee835b3076a16a7dc0382027b7683495d4be467b0214a28"
Nov 04 01:28:16.720681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.708195    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:28:16.728380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.728325    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq" event={"ID":"94d933d6-8278-432e-bdf4-7bbd5d63f8b7","Type":"ContainerDied","Data":"46536e0364602e900bd29c12890e4d6da92984d51a7492eef5cb8e1448c6abd8"}
Nov 04 01:28:16.728573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.728555    5797 scope.go:122] "RemoveContainer" containerID="223684cc34028beb4d97c39d2caf26e1029ae0456a40a7c9dd418a4ccc6af764"
Nov 04 01:28:16.866024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.865631    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.865706    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.865737    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.865772    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.865864    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.865925    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.866071    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.866163    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.866481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.866243    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo\") pod \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\" (UID: \"94d933d6-8278-432e-bdf4-7bbd5d63f8b7\") "
Nov 04 01:28:16.882167 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.882127    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:16.882693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.882668    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:16.883192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.883171    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:16.883551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.883533    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:16.910118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.909195    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:16.910118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.909288    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:16.910118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.909355    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 04 01:28:16.910118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.909411    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 04 01:28:16.931818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.931438    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "kube-api-access-2d4lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:16.938288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.937844    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:16.946525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.939338    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 04 01:28:16.946525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.939393    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 04 01:28:16.946525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.939489    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 04 01:28:16.946525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.939513    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 04 01:28:16.997028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:16.994439    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token" pod "94d933d6-8278-432e-bdf4-7bbd5d63f8b7" (UID: "94d933d6-8278-432e-bdf4-7bbd5d63f8b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038666    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038708    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d4lx\" (UniqueName: \"kubernetes.io/projected/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-kube-api-access-2d4lx\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038727    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038750    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038765    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038781    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038795    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038814    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.039025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.038830    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d933d6-8278-432e-bdf4-7bbd5d63f8b7-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:17.135054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.134966    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:28:17.786664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.786572    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"
Nov 04 01:28:17.882107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.881650    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 04 01:28:17.882107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.881711    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 04 01:28:17.910021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.906519    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 04 01:28:17.910021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.906600    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 04 01:28:17.910021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.907165    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 04 01:28:17.910021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.907202    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 04 01:28:17.940050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.936175    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:17.962045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.952044    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"]
Nov 04 01:28:17.962045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:17.953661    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3bf4f9-openshift-default-648c9d65c-ndtrq"]
Nov 04 01:28:17.971024 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:17.966651    5797 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d933d6_8278_432e_bdf4_7bbd5d63f8b7.slice/crio-46536e0364602e900bd29c12890e4d6da92984d51a7492eef5cb8e1448c6abd8\": RecentStats: unable to find data in memory cache]"
Nov 04 01:28:18.334113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.334076    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:18.539574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.539545    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" path="/var/lib/kubelet/pods/94d933d6-8278-432e-bdf4-7bbd5d63f8b7/volumes"
Nov 04 01:28:18.540878 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.540830    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:18.889830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.889793    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 04 01:28:18.890460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.890427    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 04 01:28:18.910133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.910097    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 04 01:28:18.913574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.913534    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 04 01:28:18.941560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.941524    5797 patch_prober.go:29] interesting pod/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 04 01:28:18.941845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:18.941818    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 04 01:28:19.134176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.134139    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:19.557934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.557896    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:19.587011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.586557    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"]
Nov 04 01:28:19.587011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.586764    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy" containerID="cri-o://206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5" gracePeriod=30
Nov 04 01:28:19.604433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.602661    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"]
Nov 04 01:28:19.604433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.602864    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy" containerID="cri-o://60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973" gracePeriod=30
Nov 04 01:28:19.611513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.610576    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"]
Nov 04 01:28:19.611513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.610763    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy" containerID="cri-o://219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5" gracePeriod=30
Nov 04 01:28:19.824102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.662978    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"]
Nov 04 01:28:19.824102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.805459    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy" containerID="cri-o://f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5" gracePeriod=30
Nov 04 01:28:19.860016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:19.825502    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:20.170235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:20.134958    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:20.362181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:20.361932    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:20.572260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:20.572117    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:28:21.231012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:21.230748    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:22.167010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:22.153853    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:22.885643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:22.882948    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:23.572804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:23.572760    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:23.784069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:23.784028    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:24.279064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:24.278930    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:25.139059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.139025    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:25.273671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.273638    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:28:25.321220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.321185    5797 generic.go:356] "Generic (PLEG): container finished" podID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerID="f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5" exitCode=0
Nov 04 01:28:25.321617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.321379    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" event={"ID":"8f544061-98e2-48f2-a6a2-b6d317169a74","Type":"ContainerDied","Data":"f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5"}
Nov 04 01:28:25.321617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.321434    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4" event={"ID":"8f544061-98e2-48f2-a6a2-b6d317169a74","Type":"ContainerDied","Data":"473848d1309de03df1b9153c2b75ac0a9e30c5285b39fdd7246cc1c1e4a99f63"}
Nov 04 01:28:25.321617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.321467    5797 scope.go:122] "RemoveContainer" containerID="f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5"
Nov 04 01:28:25.321794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.321631    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"
Nov 04 01:28:25.333854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.333822    5797 scope.go:122] "RemoveContainer" containerID="f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5"
Nov 04 01:28:25.334190 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:25.334158    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5\": container with ID starting with f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5 not found: ID does not exist" containerID="f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5"
Nov 04 01:28:25.334295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.334202    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5"} err="failed to get container status \"f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5\": rpc error: code = NotFound desc = could not find container \"f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5\": container with ID starting with f4474bfc2498b92f08a09e9c73d6a9df94b59a50b4cf031db5d658f9ac380fd5 not found: ID does not exist"
Nov 04 01:28:25.378951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.378918    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379241    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:25.379379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379300    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379433    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379471    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379524    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379582    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379620    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379648    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379777 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379686    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert\") pod \"8f544061-98e2-48f2-a6a2-b6d317169a74\" (UID: \"8f544061-98e2-48f2-a6a2-b6d317169a74\") "
Nov 04 01:28:25.379918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.379896    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:25.380337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.380316    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:25.380579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.380554    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:25.380670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.380562    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.380670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.380608    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544061-98e2-48f2-a6a2-b6d317169a74-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.380670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.380628    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.380670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.380669    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:25.382685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.382663    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:25.382890 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.382869    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:25.383024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.382970    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "kube-api-access-74r8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:25.383090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.383046    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo" pod "8f544061-98e2-48f2-a6a2-b6d317169a74" (UID: "8f544061-98e2-48f2-a6a2-b6d317169a74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:25.482248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.482211    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.482248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.482241    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74r8b\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-kube-api-access-74r8b\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.482248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.482261    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.482491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.482273    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.482491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.482285    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.482491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.482298    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544061-98e2-48f2-a6a2-b6d317169a74-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:25.650534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.650491    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"]
Nov 04 01:28:25.655046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.655017    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-q7lq4"]
Nov 04 01:28:25.777721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:25.777690    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:26.261762 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:26.261725    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:28:26.350338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:26.350302    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:26.630065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:26.619561    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" path="/var/lib/kubelet/pods/8f544061-98e2-48f2-a6a2-b6d317169a74/volumes"
Nov 04 01:28:26.645315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:26.645281    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:27.339838 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.339801    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:27.579366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.578663    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:27.771659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.771149    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.839534    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.839661    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.839739    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.839801    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.839832    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.839894    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.840008    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.840050    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.840136    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs\") pod \"8cded768-46fe-41cb-a166-7b435f6836b3\" (UID: \"8cded768-46fe-41cb-a166-7b435f6836b3\") "
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.863167    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.863772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.863334    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.868691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.868076    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.868691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.868403    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:27.868906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.868797    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:27.869184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.869121    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.874022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.871919    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.874022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.873786    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:27.874662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.874636    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:27.874774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.874676    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9" pod "8cded768-46fe-41cb-a166-7b435f6836b3" (UID: "8cded768-46fe-41cb-a166-7b435f6836b3"). InnerVolumeSpecName "kube-api-access-d6lr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:27.889395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.889123    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:27.904840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.904817    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:27.941318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941274    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941464    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941513    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941583    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941645    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941678    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941746    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941833    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.941934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.941890    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert\") pod \"0ba89adb-97d7-404f-958c-d06cd49a8b68\" (UID: \"0ba89adb-97d7-404f-958c-d06cd49a8b68\") "
Nov 04 01:28:27.942121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942089    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.942292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942250    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.942395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942370    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:27.942493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942474    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.942661 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942639    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.942936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942911    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942938    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942948    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942958    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cded768-46fe-41cb-a166-7b435f6836b3-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942967    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lr9\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-kube-api-access-d6lr9\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.942977    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943000    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943010    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943018    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943026    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cded768-46fe-41cb-a166-7b435f6836b3-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943035    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cded768-46fe-41cb-a166-7b435f6836b3-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943044    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943052    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba89adb-97d7-404f-958c-d06cd49a8b68-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.943510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.943060    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cded768-46fe-41cb-a166-7b435f6836b3-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:27.944411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.944278    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:27.944411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.944373    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:27.945220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.945196    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "kube-api-access-22q5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:27.946152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.946127    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token" pod "0ba89adb-97d7-404f-958c-d06cd49a8b68" (UID: "0ba89adb-97d7-404f-958c-d06cd49a8b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:27.991322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:27.991297    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:28.043943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.043909    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.043982    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044052    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044128    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044246 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044218    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044306    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044352    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044412    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044537 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044453    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:28.044599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044538    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo\") pod \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\" (UID: \"92bb280f-0dc4-4e9e-b443-1349080c5a2e\") "
Nov 04 01:28:28.044671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044642    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:28.044731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044678    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:28.044905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.044881    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:28.045128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045105    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:28.045264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045174    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045196    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045214    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045228    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045242    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22q5f\" (UniqueName: \"kubernetes.io/projected/0ba89adb-97d7-404f-958c-d06cd49a8b68-kube-api-access-22q5f\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045264 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045256    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba89adb-97d7-404f-958c-d06cd49a8b68-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045269    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.045605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.045282    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.047586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.047563    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "kube-api-access-dtvf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:28.047700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.047562    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:28:28.047700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.047666    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:28:28.048026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.048008    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token" pod "92bb280f-0dc4-4e9e-b443-1349080c5a2e" (UID: "92bb280f-0dc4-4e9e-b443-1349080c5a2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:28.146615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.146416    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.146615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.146453    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtvf4\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-kube-api-access-dtvf4\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.146615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.146469    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.146615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.146482    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.146615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.146510    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92bb280f-0dc4-4e9e-b443-1349080c5a2e-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:28.372925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.372888    5797 generic.go:356] "Generic (PLEG): container finished" podID="8cded768-46fe-41cb-a166-7b435f6836b3" containerID="219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5" exitCode=0
Nov 04 01:28:28.373423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.373391    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" event={"ID":"8cded768-46fe-41cb-a166-7b435f6836b3","Type":"ContainerDied","Data":"219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5"}
Nov 04 01:28:28.373724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.373699    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz" event={"ID":"8cded768-46fe-41cb-a166-7b435f6836b3","Type":"ContainerDied","Data":"534557b58b5beafcd94182d9e7542c6348431e3ad3b2c88abdc540e84b90da3b"}
Nov 04 01:28:28.373873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.373860    5797 scope.go:122] "RemoveContainer" containerID="219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5"
Nov 04 01:28:28.374135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.374119    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.423962    5797 generic.go:356] "Generic (PLEG): container finished" podID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerID="60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973" exitCode=0
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.424225    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.424431    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" event={"ID":"0ba89adb-97d7-404f-958c-d06cd49a8b68","Type":"ContainerDied","Data":"60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973"}
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.424491    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw" event={"ID":"0ba89adb-97d7-404f-958c-d06cd49a8b68","Type":"ContainerDied","Data":"38bdb5d83b37ae4e4385f7ebf5714f98ce0c6ae12e6a2a21973d35dad49cde37"}
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.427289    5797 generic.go:356] "Generic (PLEG): container finished" podID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerID="206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5" exitCode=0
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.427363    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" event={"ID":"92bb280f-0dc4-4e9e-b443-1349080c5a2e","Type":"ContainerDied","Data":"206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5"}
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.427408    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb" event={"ID":"92bb280f-0dc4-4e9e-b443-1349080c5a2e","Type":"ContainerDied","Data":"a21a9e9c82fba9e03663217d8d92d291c6cd17b907fafb43032f75a78369951e"}
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.427482    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.429232    5797 scope.go:122] "RemoveContainer" containerID="219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5"
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:28.429518    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5\": container with ID starting with 219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5 not found: ID does not exist" containerID="219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5"
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.429553    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5"} err="failed to get container status \"219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5\": rpc error: code = NotFound desc = could not find container \"219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5\": container with ID starting with 219853299d237bda71a46780ce46cd738603d006e3be2ac1827c72d19726b1a5 not found: ID does not exist"
Nov 04 01:28:28.438072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.429578    5797 scope.go:122] "RemoveContainer" containerID="60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973"
Nov 04 01:28:28.444486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.444467    5797 scope.go:122] "RemoveContainer" containerID="60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973"
Nov 04 01:28:28.444825 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:28.444805    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973\": container with ID starting with 60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973 not found: ID does not exist" containerID="60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973"
Nov 04 01:28:28.444892 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.444831    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973"} err="failed to get container status \"60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973\": rpc error: code = NotFound desc = could not find container \"60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973\": container with ID starting with 60f84a97ff818264d16f1fc18cc416016e29539a2166dae0300137d76b1b5973 not found: ID does not exist"
Nov 04 01:28:28.444892 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.444847    5797 scope.go:122] "RemoveContainer" containerID="206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5"
Nov 04 01:28:28.457695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.457665    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"]
Nov 04 01:28:28.459184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.459166    5797 scope.go:122] "RemoveContainer" containerID="206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5"
Nov 04 01:28:28.459536 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:28.459514    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5\": container with ID starting with 206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5 not found: ID does not exist" containerID="206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5"
Nov 04 01:28:28.459611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.459545    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5"} err="failed to get container status \"206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5\": rpc error: code = NotFound desc = could not find container \"206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5\": container with ID starting with 206016ff21c0d4c94bf00717efd7b4ce0ce1ce5f55f145971d213bcbf723f6f5 not found: ID does not exist"
Nov 04 01:28:28.461195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.461172    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-nxmgz"]
Nov 04 01:28:28.483952 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.483914    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" path="/var/lib/kubelet/pods/8cded768-46fe-41cb-a166-7b435f6836b3/volumes"
Nov 04 01:28:28.484396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.484376    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"]
Nov 04 01:28:28.484472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.484418    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-k9xlw"]
Nov 04 01:28:28.489280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.489255    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"]
Nov 04 01:28:28.495112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.495088    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-875528-openshift-default-57bd759bd9-h5pnb"]
Nov 04 01:28:28.667032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.666926    5797 scope.go:122] "RemoveContainer" containerID="5dcc25ae34d721f699fe693467a9b29bca81dc4a7f3407603fe3337c953e596d"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.876616    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"]
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877516    5797 cpu_manager.go:414] "containerMap: removing container" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877540    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877559    5797 cpu_manager.go:414] "containerMap: removing container" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877568    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877581    5797 cpu_manager.go:414] "containerMap: removing container" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877589    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877623    5797 cpu_manager.go:414] "containerMap: removing container" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877631    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877643    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877649    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877682    5797 cpu_manager.go:414] "containerMap: removing container" podUID="70f384bd-68ce-4fd5-8da5-8b5197924582" containerName="revoke-keys"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877689    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f384bd-68ce-4fd5-8da5-8b5197924582" containerName="revoke-keys"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877840    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877852    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f544061-98e2-48f2-a6a2-b6d317169a74" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877863    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877877    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cded768-46fe-41cb-a166-7b435f6836b3" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877887    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" containerName="istio-proxy"
Nov 04 01:28:28.879694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.877900    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f384bd-68ce-4fd5-8da5-8b5197924582" containerName="revoke-keys"
Nov 04 01:28:28.887562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.886287    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:28.892678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.891903    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:28:28.892678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.892626    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:28:28.902021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.901728    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"]
Nov 04 01:28:28.926587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.926502    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:28.967710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.967647    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:28.967972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.967947    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:28.968149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:28.968128    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.069410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.069372    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.069605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.069448    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.069605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.069513    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.070301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.070270    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.071939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.071914    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.087728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.087698    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7\") pod \"maas-api-revoke-sub-3e3562ed-thz9v\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.252066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.246219    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:29.603185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.603153    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"]
Nov 04 01:28:29.606104 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:28:29.606072    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955d5da8_8e4c_4f3f_8101_ffa4792083da.slice/crio-5651e4f6d565c09a8e7e40ae015ed7618ac48f0eceee75a0d7c4f93f1ebf7772 WatchSource:0}: Error finding container 5651e4f6d565c09a8e7e40ae015ed7618ac48f0eceee75a0d7c4f93f1ebf7772: Status 404 returned error can't find the container with id 5651e4f6d565c09a8e7e40ae015ed7618ac48f0eceee75a0d7c4f93f1ebf7772
Nov 04 01:28:29.919014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:29.917372    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:30.488116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.488076    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba89adb-97d7-404f-958c-d06cd49a8b68" path="/var/lib/kubelet/pods/0ba89adb-97d7-404f-958c-d06cd49a8b68/volumes"
Nov 04 01:28:30.488757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.488741    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92bb280f-0dc4-4e9e-b443-1349080c5a2e" path="/var/lib/kubelet/pods/92bb280f-0dc4-4e9e-b443-1349080c5a2e/volumes"
Nov 04 01:28:30.489159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.489140    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v" event={"ID":"955d5da8-8e4c-4f3f-8101-ffa4792083da","Type":"ContainerStarted","Data":"bbdc90c4d0b5bd2226abb325af884502e9c76aa12ccdc090b9671bd68514b95e"}
Nov 04 01:28:30.489207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.489194    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v" event={"ID":"955d5da8-8e4c-4f3f-8101-ffa4792083da","Type":"ContainerStarted","Data":"5651e4f6d565c09a8e7e40ae015ed7618ac48f0eceee75a0d7c4f93f1ebf7772"}
Nov 04 01:28:30.505904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.505863    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v" podStartSLOduration=2.5058513959999997 podStartE2EDuration="2.505851396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:30.503702897 +0000 UTC m=+1498.145609149" watchObservedRunningTime="2026-11-04 01:28:30.505851396 +0000 UTC m=+1498.147757660"
Nov 04 01:28:30.742440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.742312    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:30.917161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.917132    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"]
Nov 04 01:28:30.989269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:30.989195    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:31.199668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:31.199634    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:31.540068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:31.539233    5797 generic.go:356] "Generic (PLEG): container finished" podID="955d5da8-8e4c-4f3f-8101-ffa4792083da" containerID="bbdc90c4d0b5bd2226abb325af884502e9c76aa12ccdc090b9671bd68514b95e" exitCode=0
Nov 04 01:28:31.540068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:31.539523    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v" event={"ID":"955d5da8-8e4c-4f3f-8101-ffa4792083da","Type":"ContainerDied","Data":"bbdc90c4d0b5bd2226abb325af884502e9c76aa12ccdc090b9671bd68514b95e"}
Nov 04 01:28:31.878280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:31.878241    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:32.083013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.079736    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:32.285874 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.285844    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:32.526703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.526619    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:32.557013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.556492    5797 generic.go:356] "Generic (PLEG): container finished" podID="1a2e54ab-acb8-4611-bf2d-534053133f77" containerID="277377df435b3a2c3074d418d2ff01b1b30f5320e301cdcd76f9450a43c2bc49" exitCode=0
Nov 04 01:28:32.561032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.557908    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" event={"ID":"1a2e54ab-acb8-4611-bf2d-534053133f77","Type":"ContainerDied","Data":"277377df435b3a2c3074d418d2ff01b1b30f5320e301cdcd76f9450a43c2bc49"}
Nov 04 01:28:32.561032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.557961    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m" event={"ID":"1a2e54ab-acb8-4611-bf2d-534053133f77","Type":"ContainerStarted","Data":"5501bc21388bfffce5f8c9a6230c450c4411f7cff3a1c4fe34ce1383cf814577"}
Nov 04 01:28:32.561032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.558030    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:28:32.830709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.830681    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:32.883637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.883596    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca\") pod \"955d5da8-8e4c-4f3f-8101-ffa4792083da\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") "
Nov 04 01:28:32.883851 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.883666    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7\") pod \"955d5da8-8e4c-4f3f-8101-ffa4792083da\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") "
Nov 04 01:28:32.883851 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.883833    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca\") pod \"955d5da8-8e4c-4f3f-8101-ffa4792083da\" (UID: \"955d5da8-8e4c-4f3f-8101-ffa4792083da\") "
Nov 04 01:28:32.884476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.884445    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca" pod "955d5da8-8e4c-4f3f-8101-ffa4792083da" (UID: "955d5da8-8e4c-4f3f-8101-ffa4792083da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:32.888855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.888827    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7" pod "955d5da8-8e4c-4f3f-8101-ffa4792083da" (UID: "955d5da8-8e4c-4f3f-8101-ffa4792083da"). InnerVolumeSpecName "kube-api-access-frqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:32.888983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.888963    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca" pod "955d5da8-8e4c-4f3f-8101-ffa4792083da" (UID: "955d5da8-8e4c-4f3f-8101-ffa4792083da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:28:32.986453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.986422    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:32.986453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.986449    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/955d5da8-8e4c-4f3f-8101-ffa4792083da-kube-api-access-frqt7\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:32.986453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:32.986461    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d5da8-8e4c-4f3f-8101-ffa4792083da-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:33.229816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:33.229775    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:33.579384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:33.579286    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v" event={"ID":"955d5da8-8e4c-4f3f-8101-ffa4792083da","Type":"ContainerDied","Data":"5651e4f6d565c09a8e7e40ae015ed7618ac48f0eceee75a0d7c4f93f1ebf7772"}
Nov 04 01:28:33.579384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:33.579349    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5651e4f6d565c09a8e7e40ae015ed7618ac48f0eceee75a0d7c4f93f1ebf7772"
Nov 04 01:28:33.579384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:33.579307    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"
Nov 04 01:28:33.999478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:33.999442    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:34.196901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:34.196864    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:34.513013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:34.510377    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:34.907846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:34.907805    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:35.296070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:35.295290    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:28:35.340686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:35.340650    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:35.581451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:35.581360    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:35.775042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:35.775006    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:35.917786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:35.917741    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" containerID="cri-o://42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1" gracePeriod=25
Nov 04 01:28:36.166377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:36.166345    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:36.881274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:36.881239    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:37.475545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:37.475510    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:37.717821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:37.717791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:37.934420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:37.934381    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:38.375023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:38.373777    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:38.658402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:38.658271    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:39.203499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:39.203446    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:39.411420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:39.411388    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:39.645633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:39.645594    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:39.651361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:39.651329    5797 patch_prober.go:29] interesting pod/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": read tcp 10.134.0.2:58410->10.134.0.166:15021: read: connection reset by peer" start-of-body=
Nov 04 01:28:39.651488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:39.651409    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": read tcp 10.134.0.2:58410->10.134.0.166:15021: read: connection reset by peer"
Nov 04 01:28:39.853733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:39.853644    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:40.229136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.229093    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:40.672409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.672375    5797 generic.go:356] "Generic (PLEG): container finished" podID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerID="f98d4d2b4f47c74e9fbb279d985f5a2a1a519d50227ade3640e8f5a83324a2b5" exitCode=0
Nov 04 01:28:40.672599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.672448    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" event={"ID":"c680caa3-cc12-4f0a-b909-cca4a3e9adcb","Type":"ContainerDied","Data":"f98d4d2b4f47c74e9fbb279d985f5a2a1a519d50227ade3640e8f5a83324a2b5"}
Nov 04 01:28:40.672908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.672893    5797 scope.go:122] "RemoveContainer" containerID="f98d4d2b4f47c74e9fbb279d985f5a2a1a519d50227ade3640e8f5a83324a2b5"
Nov 04 01:28:40.674634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.674602    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:40.674766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.674684    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:40.674766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.674716    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:28:40.700904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:40.700847    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:41.244559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:41.243084    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:41.438186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:41.438148    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:41.638799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:41.638768    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:41.682432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:41.682392    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" event={"ID":"c680caa3-cc12-4f0a-b909-cca4a3e9adcb","Type":"ContainerStarted","Data":"7f72c477f031ae85efabd72990a7b9a305341e3a6d3cf2dbf6dca6654a151bb5"}
Nov 04 01:28:42.270846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:42.270808    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:42.381042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:42.375265    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:28:42.393392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:42.393359    5797 patch_prober.go:29] interesting pod/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 04 01:28:42.393674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:42.393642    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 04 01:28:42.694939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:42.694906    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:42.901805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:42.901772    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:43.123700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:43.123649    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:43.371792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:43.371761    5797 patch_prober.go:29] interesting pod/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 04 01:28:43.373863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:43.373792    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 04 01:28:43.374384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:43.373611    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:44.336768 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:44.336736    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:44.378423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:44.378365    5797 patch_prober.go:29] interesting pod/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 04 01:28:44.378906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:44.378421    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 04 01:28:44.539833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:44.539802    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:44.781420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:44.781381    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:45.300614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.300583    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:45.379898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.379861    5797 patch_prober.go:29] interesting pod/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 04 01:28:45.380499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.380469    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 04 01:28:45.492897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.490589    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 01:28:45.504621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.504591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:45.713164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.713127    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:45.989132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:45.989040    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:46.373030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:46.372957    5797 patch_prober.go:29] interesting pod/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 04 01:28:46.373320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:46.373291    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 04 01:28:46.513059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:46.513032    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:28:46.637061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:46.636969    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:28:46.825278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:46.825119    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:47.119227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:47.119199    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:47.376061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:47.375976    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:28:47.376273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:47.376247    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:28:47.377044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:47.377024    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:28:47.462696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:47.462645    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:47.789727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:47.789636    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:48.231414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:48.231375    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:48.433172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:48.433139    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:49.087503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:49.087466    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:49.656183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:49.656141    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:49.864940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:49.864906    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:50.609009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:50.606544    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:51.613011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:51.611577    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:52.605908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:52.605874    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:53.503499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:53.503455    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:54.157275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:54.157164    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:54.440148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:54.440053    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:54.647764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:54.647210    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:54.848841 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:54.848762    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:55.285529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:55.285493    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:55.296393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:55.296364    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:28:55.300758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:55.300729    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bb86d5d94-r4w4m"
Nov 04 01:28:55.490151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:55.490113    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 01:28:55.491391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:55.491358    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:56.053108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.053075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:56.203386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.203363    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:28:56.252639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.252608    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl\" (UniqueName: \"kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl\") pod \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") "
Nov 04 01:28:56.252816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.252713    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume\") pod \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\" (UID: \"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f\") "
Nov 04 01:28:56.253460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.253427    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume" pod "dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" (UID: "dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:28:56.254945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.254908    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl" pod "dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" (UID: "dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f"). InnerVolumeSpecName "kube-api-access-qmrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:28:56.259384 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:56.259355    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:28:56.355286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.355256    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrhl\" (UniqueName: \"kubernetes.io/projected/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-kube-api-access-qmrhl\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:56.355286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.355285    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:28:56.879534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.879468    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:56.889974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.889945    5797 generic.go:356] "Generic (PLEG): container finished" podID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerID="42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1" exitCode=0
Nov 04 01:28:56.890195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.890030    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" event={"ID":"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f","Type":"ContainerDied","Data":"42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1"}
Nov 04 01:28:56.890195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.890086    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9" event={"ID":"dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f","Type":"ContainerDied","Data":"796ad0aef49b50892d631be1390be1f6a64cc20b8dc11e3ad2b8efe5bfa146db"}
Nov 04 01:28:56.890195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.890107    5797 scope.go:122] "RemoveContainer" containerID="42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1"
Nov 04 01:28:56.890195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.890043    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"
Nov 04 01:28:56.901718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.901700    5797 scope.go:122] "RemoveContainer" containerID="42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1"
Nov 04 01:28:56.902058 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:28:56.902036    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1\": container with ID starting with 42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1 not found: ID does not exist" containerID="42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1"
Nov 04 01:28:56.902144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.902066    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1"} err="failed to get container status \"42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1\": rpc error: code = NotFound desc = could not find container \"42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1\": container with ID starting with 42a4da77fd9cc8e17312e6628095cc86fa8b6a73dc0f5b8fbc4bf46587cb6ff1 not found: ID does not exist"
Nov 04 01:28:56.910085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.909427    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"]
Nov 04 01:28:56.913524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:56.913502    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ba844c-86cd46fc55-q6vw9"]
Nov 04 01:28:57.131648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:57.131565    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:57.662914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:57.662884    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:58.224977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:58.224942    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:58.480289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:58.480183    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" path="/var/lib/kubelet/pods/dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f/volumes"
Nov 04 01:28:58.511050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:58.511017    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:58.720953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:58.720911    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:58.915629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:58.915595    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:59.155588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:59.155541    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:59.369085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:59.367913    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:59.569761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:59.569730    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:28:59.774545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:28:59.774454    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:00.010525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.010488    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:00.013191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.013165    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"]
Nov 04 01:29:00.013603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.013564    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy" containerID="cri-o://3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151" gracePeriod=30
Nov 04 01:29:00.022324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.022295    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"]
Nov 04 01:29:00.022588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.022561    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy" containerID="cri-o://16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741" gracePeriod=30
Nov 04 01:29:00.025044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.024947    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"]
Nov 04 01:29:00.025241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.025211    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy" containerID="cri-o://2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e" gracePeriod=30
Nov 04 01:29:00.225276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.225241    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:00.447664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.431174    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:00.644250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:00.644055    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:01.099922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:01.099886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:01.296824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:01.296794    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:01.501493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:01.501462    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:01.890746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:01.890712    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:02.590207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:02.590172    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:03.474488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:03.474447    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:04.191205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:04.191170    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:04.401367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:04.401333    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:05.044727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.044696    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:05.341550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.341519    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:29:05.423815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.423345    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:29:05.429283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.429251    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:29:05.432138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.432112    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:05.437609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437579    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.437799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437639    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.437799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437724    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.437799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437755    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.438027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437848    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.438027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437870    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.438027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.437978    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.438027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438006    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.438257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438055    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.438257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438181    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.438257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438209    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.438429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438295    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:29:05.438429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438341    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8\") pod \"24765e2f-e83c-4abb-92e2-2d04ec59899a\" (UID: \"24765e2f-e83c-4abb-92e2-2d04ec59899a\") "
Nov 04 01:29:05.438547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.438507    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.439104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.439060    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24765e2f-e83c-4abb-92e2-2d04ec59899a-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.439104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.439088    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.439104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.439103    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.439317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.439117    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.439317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.439130    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.441078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.441045    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:29:05.441195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.441088    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.441302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.441287    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:05.441543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.441517    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8" pod "24765e2f-e83c-4abb-92e2-2d04ec59899a" (UID: "24765e2f-e83c-4abb-92e2-2d04ec59899a"). InnerVolumeSpecName "kube-api-access-dwhd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545268    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545346    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545380    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545419    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545459    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545518    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545580    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545631    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545671    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545701    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545755    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545800    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545829    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545864    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545894    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.545931    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546025    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket\") pod \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\" (UID: \"07da5e23-0dfd-4c32-80b2-3b730e9bce8b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546101    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert\") pod \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\" (UID: \"495c9d92-defe-4156-9c45-8e6ed42e7b5b\") "
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546424    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546785    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546808    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546823    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhd8\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-kube-api-access-dwhd8\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546838    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.546852    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24765e2f-e83c-4abb-92e2-2d04ec59899a-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.557077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.552670    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.571135    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.571346    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.571669    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.572523    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.572731    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.572781    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.572825    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.580491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.579883    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:29:05.598254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.598175    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:29:05.601442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.601407    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "kube-api-access-mzvrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:05.601708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.601689    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "kube-api-access-sllhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:05.611418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.611264    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.634344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.632861    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:05.634344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.632931    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:29:05.638971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.638933    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token" pod "495c9d92-defe-4156-9c45-8e6ed42e7b5b" (UID: "495c9d92-defe-4156-9c45-8e6ed42e7b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:05.639405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.639034    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:05.640560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.640402    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy" pod "07da5e23-0dfd-4c32-80b2-3b730e9bce8b" (UID: "07da5e23-0dfd-4c32-80b2-3b730e9bce8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:05.647638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647612    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647635    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sllhc\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-kube-api-access-sllhc\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647646    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647656    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647664    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzvrw\" (UniqueName: \"kubernetes.io/projected/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-kube-api-access-mzvrw\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647672    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647680    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647688    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647696    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647705    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647713    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647720    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647728    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647735    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495c9d92-defe-4156-9c45-8e6ed42e7b5b-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647743    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495c9d92-defe-4156-9c45-8e6ed42e7b5b-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647751    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.647826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.647761    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07da5e23-0dfd-4c32-80b2-3b730e9bce8b-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:05.979240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.979200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:05.980905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.980877    5797 generic.go:356] "Generic (PLEG): container finished" podID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerID="3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151" exitCode=0
Nov 04 01:29:05.981101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.980921    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"
Nov 04 01:29:05.981101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.980949    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" event={"ID":"24765e2f-e83c-4abb-92e2-2d04ec59899a","Type":"ContainerDied","Data":"3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151"}
Nov 04 01:29:05.981101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.981009    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj" event={"ID":"24765e2f-e83c-4abb-92e2-2d04ec59899a","Type":"ContainerDied","Data":"418f35f3dac9908bd770de8a2a93638f1809a10f76a80328d0548475472dc3a9"}
Nov 04 01:29:05.981101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.981040    5797 scope.go:122] "RemoveContainer" containerID="3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151"
Nov 04 01:29:05.982464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.982442    5797 generic.go:356] "Generic (PLEG): container finished" podID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerID="2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e" exitCode=0
Nov 04 01:29:05.982576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.982483    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"
Nov 04 01:29:05.982576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.982523    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" event={"ID":"07da5e23-0dfd-4c32-80b2-3b730e9bce8b","Type":"ContainerDied","Data":"2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e"}
Nov 04 01:29:05.982678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.982575    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5" event={"ID":"07da5e23-0dfd-4c32-80b2-3b730e9bce8b","Type":"ContainerDied","Data":"2f883f8e7f157be7d04552c7d43bfc354018e163fcd5cc7744d7acc23bb498f8"}
Nov 04 01:29:05.984268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.984198    5797 generic.go:356] "Generic (PLEG): container finished" podID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerID="16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741" exitCode=0
Nov 04 01:29:05.984268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.984232    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"
Nov 04 01:29:05.984268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.984235    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" event={"ID":"495c9d92-defe-4156-9c45-8e6ed42e7b5b","Type":"ContainerDied","Data":"16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741"}
Nov 04 01:29:05.984510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.984283    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9" event={"ID":"495c9d92-defe-4156-9c45-8e6ed42e7b5b","Type":"ContainerDied","Data":"33670b22d57e05511f97c23a90e42c2f53247368377d9a712b3788fe1f38f95d"}
Nov 04 01:29:05.992855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.992833    5797 scope.go:122] "RemoveContainer" containerID="3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151"
Nov 04 01:29:05.993127 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:29:05.993106    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151\": container with ID starting with 3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151 not found: ID does not exist" containerID="3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151"
Nov 04 01:29:05.993222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.993136    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151"} err="failed to get container status \"3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151\": rpc error: code = NotFound desc = could not find container \"3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151\": container with ID starting with 3fe20a18f1c7fa5e63dc847813fb1174ca0781bfc4520d2a686daa63f095a151 not found: ID does not exist"
Nov 04 01:29:05.993222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:05.993159    5797 scope.go:122] "RemoveContainer" containerID="2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e"
Nov 04 01:29:06.003484 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.003465    5797 scope.go:122] "RemoveContainer" containerID="2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e"
Nov 04 01:29:06.003786 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:29:06.003761    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e\": container with ID starting with 2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e not found: ID does not exist" containerID="2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e"
Nov 04 01:29:06.003893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.003791    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e"} err="failed to get container status \"2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e\": rpc error: code = NotFound desc = could not find container \"2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e\": container with ID starting with 2573285fc9091d79ab9e482908c498456d397bc6cb832a877fee372abfa7d04e not found: ID does not exist"
Nov 04 01:29:06.003893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.003806    5797 scope.go:122] "RemoveContainer" containerID="16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741"
Nov 04 01:29:06.011328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.011307    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"]
Nov 04 01:29:06.014599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.014561    5797 scope.go:122] "RemoveContainer" containerID="16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741"
Nov 04 01:29:06.014950 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:29:06.014908    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741\": container with ID starting with 16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741 not found: ID does not exist" containerID="16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741"
Nov 04 01:29:06.015051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.014957    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741"} err="failed to get container status \"16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741\": rpc error: code = NotFound desc = could not find container \"16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741\": container with ID starting with 16f1cb19d50df6bbe46c06126d3d366afa210fb5c2ba34359d900930adc03741 not found: ID does not exist"
Nov 04 01:29:06.016504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.016486    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-vb8hj"]
Nov 04 01:29:06.028898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.028877    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"]
Nov 04 01:29:06.034382 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.034359    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-px2v9"]
Nov 04 01:29:06.045214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.045191    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"]
Nov 04 01:29:06.050134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.050109    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ba844c-openshift-default-549db56fd5-5dck5"]
Nov 04 01:29:06.479514 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.479480    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" path="/var/lib/kubelet/pods/07da5e23-0dfd-4c32-80b2-3b730e9bce8b/volumes"
Nov 04 01:29:06.480250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.480226    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" path="/var/lib/kubelet/pods/24765e2f-e83c-4abb-92e2-2d04ec59899a/volumes"
Nov 04 01:29:06.480799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.480780    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" path="/var/lib/kubelet/pods/495c9d92-defe-4156-9c45-8e6ed42e7b5b/volumes"
Nov 04 01:29:06.938477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:06.938438    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:07.171624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:07.171585    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:07.381417 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:07.381381    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:07.914013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:07.906443    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:08.328575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:08.328431    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:08.526704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:08.526663    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:08.722549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:08.722501    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:08.950306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:08.943455    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:09.584755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:09.584716    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:09.895831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:09.895796    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:10.575855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:10.575820    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:11.139746 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:11.139715    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:11.412157 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:11.391579    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:11.608336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:11.608295    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:11.806391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:11.806293    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:12.278562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:12.278525    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:13.076923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:13.076892    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:13.777074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:13.777039    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:14.355607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:14.355571    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:14.699256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:14.699222    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:14.902117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:14.902082    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:15.364594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:15.364562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:15.944187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:15.944152    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:16.137657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:16.137621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:16.412488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:16.412452    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:16.824551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:16.824426    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:17.102303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:17.102200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:17.673670 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:17.673637    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:17.899522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:17.899484    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:18.146664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:18.146605    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:18.349562 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:18.349523    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:18.802014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:18.798792    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:19.748958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:19.748896    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:20.564014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:20.563968    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:21.084792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:21.084754    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:21.282624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:21.282591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:21.495129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:21.495076    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:21.743588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:21.743553    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:21.920457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:21.920419    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"]
Nov 04 01:29:21.921013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:21.920937    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy" containerID="cri-o://7f72c477f031ae85efabd72990a7b9a305341e3a6d3cf2dbf6dca6654a151bb5" gracePeriod=30
Nov 04 01:29:22.073169 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.073091    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:22.596796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.596756    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:22.839038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.835649    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:22.934834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.934798    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"]
Nov 04 01:29:22.935366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935350    5797 cpu_manager.go:414] "containerMap: removing container" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935368    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935403    5797 cpu_manager.go:414] "containerMap: removing container" podUID="955d5da8-8e4c-4f3f-8101-ffa4792083da" containerName="revoke-keys"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935408    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955d5da8-8e4c-4f3f-8101-ffa4792083da" containerName="revoke-keys"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935416    5797 cpu_manager.go:414] "containerMap: removing container" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935420    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935437    5797 cpu_manager.go:414] "containerMap: removing container" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy"
Nov 04 01:29:22.935440 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935442    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935451    5797 cpu_manager.go:414] "containerMap: removing container" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935456    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935470    5797 cpu_manager.go:414] "containerMap: removing container" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935474    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935572    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="955d5da8-8e4c-4f3f-8101-ffa4792083da" containerName="revoke-keys"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935580    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="495c9d92-defe-4156-9c45-8e6ed42e7b5b" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935588    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="07da5e23-0dfd-4c32-80b2-3b730e9bce8b" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935594    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="24765e2f-e83c-4abb-92e2-2d04ec59899a" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935600    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d933d6-8278-432e-bdf4-7bbd5d63f8b7" containerName="istio-proxy"
Nov 04 01:29:22.935662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.935605    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6c2f2e-3bc7-4f8e-8102-ad52576f8a4f" containerName="payload-pre-processing"
Nov 04 01:29:22.938951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.938929    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:22.945154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.945131    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:29:22.949855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.946105    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:29:22.950400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:22.950383    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"]
Nov 04 01:29:23.039328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.039293    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.039495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.039340    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgmx\" (UniqueName: \"kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.039495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.039426    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.140728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.140698    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgmx\" (UniqueName: \"kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.140913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.140746    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.140983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.140962    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.141575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.141549    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.143275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.143257    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.152858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.152837    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgmx\" (UniqueName: \"kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx\") pod \"maas-api-revoke-sub-17cbaa93-nxkl9\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.258269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.258177    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:23.562545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.562508    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"]
Nov 04 01:29:23.564736 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:29:23.564692    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7151fd7a_5b3c_4001_ad42_cdb515c1425b.slice/crio-75fb82974dab0b288741b1230d427256ea4d77e4811d14f129525e605dea3b31 WatchSource:0}: Error finding container 75fb82974dab0b288741b1230d427256ea4d77e4811d14f129525e605dea3b31: Status 404 returned error can't find the container with id 75fb82974dab0b288741b1230d427256ea4d77e4811d14f129525e605dea3b31
Nov 04 01:29:23.572790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:23.572762    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:24.207173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:24.207126    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9" event={"ID":"7151fd7a-5b3c-4001-ad42-cdb515c1425b","Type":"ContainerStarted","Data":"4cd19bf74b175f7aa1a5052f0d8bba51c773f5686aec9b3857dd8c71eec23739"}
Nov 04 01:29:24.207637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:24.207204    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9" event={"ID":"7151fd7a-5b3c-4001-ad42-cdb515c1425b","Type":"ContainerStarted","Data":"75fb82974dab0b288741b1230d427256ea4d77e4811d14f129525e605dea3b31"}
Nov 04 01:29:24.228034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:24.227974    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9" podStartSLOduration=2.227958019 podStartE2EDuration="2.227958019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:24.22533782 +0000 UTC m=+1551.867244090" watchObservedRunningTime="2026-11-04 01:29:24.227958019 +0000 UTC m=+1551.869864268"
Nov 04 01:29:24.429384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:24.429303    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:25.079069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:25.079030    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:25.215608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:25.215576    5797 generic.go:356] "Generic (PLEG): container finished" podID="7151fd7a-5b3c-4001-ad42-cdb515c1425b" containerID="4cd19bf74b175f7aa1a5052f0d8bba51c773f5686aec9b3857dd8c71eec23739" exitCode=0
Nov 04 01:29:25.216088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:25.215663    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9" event={"ID":"7151fd7a-5b3c-4001-ad42-cdb515c1425b","Type":"ContainerDied","Data":"4cd19bf74b175f7aa1a5052f0d8bba51c773f5686aec9b3857dd8c71eec23739"}
Nov 04 01:29:25.775376 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:25.775342    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:26.256425 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:29:26.256383    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:29:26.389950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.389923    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:26.480307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.480211    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:26.488299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.487883    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca\") pod \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") "
Nov 04 01:29:26.488299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.488063    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx\" (UniqueName: \"kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx\") pod \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") "
Nov 04 01:29:26.488299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.488119    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca\") pod \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\" (UID: \"7151fd7a-5b3c-4001-ad42-cdb515c1425b\") "
Nov 04 01:29:26.488299 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.488263    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca" pod "7151fd7a-5b3c-4001-ad42-cdb515c1425b" (UID: "7151fd7a-5b3c-4001-ad42-cdb515c1425b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:29:26.488699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.488676    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:26.493177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.493148    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx" pod "7151fd7a-5b3c-4001-ad42-cdb515c1425b" (UID: "7151fd7a-5b3c-4001-ad42-cdb515c1425b"). InnerVolumeSpecName "kube-api-access-tdgmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:26.498143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.498112    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca" pod "7151fd7a-5b3c-4001-ad42-cdb515c1425b" (UID: "7151fd7a-5b3c-4001-ad42-cdb515c1425b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:29:26.590088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.590059    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgmx\" (UniqueName: \"kubernetes.io/projected/7151fd7a-5b3c-4001-ad42-cdb515c1425b-kube-api-access-tdgmx\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:26.590088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:26.590086    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7151fd7a-5b3c-4001-ad42-cdb515c1425b-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.261648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.261550    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9" event={"ID":"7151fd7a-5b3c-4001-ad42-cdb515c1425b","Type":"ContainerDied","Data":"75fb82974dab0b288741b1230d427256ea4d77e4811d14f129525e605dea3b31"}
Nov 04 01:29:27.261648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.261610    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75fb82974dab0b288741b1230d427256ea4d77e4811d14f129525e605dea3b31"
Nov 04 01:29:27.262240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.261686    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"
Nov 04 01:29:27.278213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.278161    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:27.280520 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.280489    5797 generic.go:356] "Generic (PLEG): container finished" podID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerID="7f72c477f031ae85efabd72990a7b9a305341e3a6d3cf2dbf6dca6654a151bb5" exitCode=0
Nov 04 01:29:27.280645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.280524    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" event={"ID":"c680caa3-cc12-4f0a-b909-cca4a3e9adcb","Type":"ContainerDied","Data":"7f72c477f031ae85efabd72990a7b9a305341e3a6d3cf2dbf6dca6654a151bb5"}
Nov 04 01:29:27.280645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.280607    5797 scope.go:122] "RemoveContainer" containerID="f98d4d2b4f47c74e9fbb279d985f5a2a1a519d50227ade3640e8f5a83324a2b5"
Nov 04 01:29:27.292091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.292065    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:29:27.454932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.454903    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.454950    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455009    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455061    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455127 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455105    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455135    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455223    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455242    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:27.455367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455297    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455360    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9\") pod \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\" (UID: \"c680caa3-cc12-4f0a-b909-cca4a3e9adcb\") "
Nov 04 01:29:27.455632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455388    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:27.455632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455505    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:27.455632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455570    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:29:27.455790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455640    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:27.455941 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455919    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.456029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455946    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.456029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455961    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.456029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455970    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.456029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.455979    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.458429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.458397    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "kube-api-access-s9bv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:27.458604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.458585    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:29:27.458672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.458583    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:27.459118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.459097    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token" pod "c680caa3-cc12-4f0a-b909-cca4a3e9adcb" (UID: "c680caa3-cc12-4f0a-b909-cca4a3e9adcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:27.460832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.460798    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"]
Nov 04 01:29:27.556891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.556802    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.556891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.556831    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.556891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.556845    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.556891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.556857    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9bv9\" (UniqueName: \"kubernetes.io/projected/c680caa3-cc12-4f0a-b909-cca4a3e9adcb-kube-api-access-s9bv9\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:27.710976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:27.710915    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:28.154062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.154024    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:28.288240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.288213    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"
Nov 04 01:29:28.288718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.288257    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n" event={"ID":"c680caa3-cc12-4f0a-b909-cca4a3e9adcb","Type":"ContainerDied","Data":"930b1b92f4e64e49c70f2603df5330600204834f65af1936d6093bf664a69c59"}
Nov 04 01:29:28.288718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.288320    5797 scope.go:122] "RemoveContainer" containerID="7f72c477f031ae85efabd72990a7b9a305341e3a6d3cf2dbf6dca6654a151bb5"
Nov 04 01:29:28.355420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.352275    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:28.371391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.369393    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"]
Nov 04 01:29:28.373453 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.373404    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6ecd1f-openshift-default-cbd7b5b8f-l798n"]
Nov 04 01:29:28.478243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.478168    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" path="/var/lib/kubelet/pods/c680caa3-cc12-4f0a-b909-cca4a3e9adcb/volumes"
Nov 04 01:29:28.552551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.552503    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:28.790486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.790401    5797 scope.go:122] "RemoveContainer" containerID="5468be55aef0f918df758a401ab84477feaae2529c3dfefdfdb04d15783d67e1"
Nov 04 01:29:28.801459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:28.801433    5797 scope.go:122] "RemoveContainer" containerID="31df58fc388c16e5b26f6cc3a15af78356624c175d8528070ff141a7a97c5994"
Nov 04 01:29:29.087141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:29.087052    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:29.559554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:29.559523    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:29.887109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:29.887068    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:30.086349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:30.086308    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:30.518896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:30.518866    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:30.716395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:30.716361    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:30.914540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:30.914510    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:31.408489 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:31.408453    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:32.086364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:32.086335    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:32.288603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:32.288571    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:32.877885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:32.877843    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:33.657961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:33.657924    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:33.875868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:33.875799    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:34.078301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.078185    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:34.587976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.587938    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"]
Nov 04 01:29:34.588766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.588742    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7151fd7a-5b3c-4001-ad42-cdb515c1425b" containerName="revoke-keys"
Nov 04 01:29:34.588835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.588770    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7151fd7a-5b3c-4001-ad42-cdb515c1425b" containerName="revoke-keys"
Nov 04 01:29:34.588873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.588835    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy"
Nov 04 01:29:34.588873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.588843    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy"
Nov 04 01:29:34.588873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.588864    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy"
Nov 04 01:29:34.588873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.588872    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy"
Nov 04 01:29:34.589049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.589022    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy"
Nov 04 01:29:34.589049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.589039    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c680caa3-cc12-4f0a-b909-cca4a3e9adcb" containerName="istio-proxy"
Nov 04 01:29:34.589115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.589052    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7151fd7a-5b3c-4001-ad42-cdb515c1425b" containerName="revoke-keys"
Nov 04 01:29:34.595212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.595180    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.598722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.598694    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"]
Nov 04 01:29:34.663513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.663474    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.663513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.663525    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.664188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.663733    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn74w\" (UniqueName: \"kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.669116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.669094    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:29:34.669237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.669127    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:29:34.764624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.764589    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn74w\" (UniqueName: \"kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.764800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.764712    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.764800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.764743    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.765487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.765464    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.767136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.767115    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.776452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.776431    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn74w\" (UniqueName: \"kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w\") pod \"maas-api-revoke-sub-a3cddec8-fsbts\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:34.885378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.885344    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:34.916334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:34.916307    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:35.201334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:35.201276    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"]
Nov 04 01:29:35.203620 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:29:35.203584    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a89f60_50d5_4e2d_8444_147d39c965e0.slice/crio-6275bc069a582e0d1d5127137cd8f027c77aadbd279ca6959f252d2bccab3d8b WatchSource:0}: Error finding container 6275bc069a582e0d1d5127137cd8f027c77aadbd279ca6959f252d2bccab3d8b: Status 404 returned error can't find the container with id 6275bc069a582e0d1d5127137cd8f027c77aadbd279ca6959f252d2bccab3d8b
Nov 04 01:29:35.282433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:35.282397    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:35.340676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:35.340645    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts" event={"ID":"49a89f60-50d5-4e2d-8444-147d39c965e0","Type":"ContainerStarted","Data":"6275bc069a582e0d1d5127137cd8f027c77aadbd279ca6959f252d2bccab3d8b"}
Nov 04 01:29:35.485371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:35.485337    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:35.700014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:35.686222    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:36.022391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:36.022355    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:36.349078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:36.348962    5797 generic.go:356] "Generic (PLEG): container finished" podID="49a89f60-50d5-4e2d-8444-147d39c965e0" containerID="bea3da2a3447d7c316136480cf6f12f978aad3f1177883ec7aa660b4550b0aa4" exitCode=0
Nov 04 01:29:36.349078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:36.349044    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts" event={"ID":"49a89f60-50d5-4e2d-8444-147d39c965e0","Type":"ContainerDied","Data":"bea3da2a3447d7c316136480cf6f12f978aad3f1177883ec7aa660b4550b0aa4"}
Nov 04 01:29:36.773676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:36.773635    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:37.693332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.693296    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:37.698751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.698728    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:37.816316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.816283    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca\") pod \"49a89f60-50d5-4e2d-8444-147d39c965e0\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") "
Nov 04 01:29:37.816766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.816412    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca\") pod \"49a89f60-50d5-4e2d-8444-147d39c965e0\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") "
Nov 04 01:29:37.816766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.816442    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w\" (UniqueName: \"kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w\") pod \"49a89f60-50d5-4e2d-8444-147d39c965e0\" (UID: \"49a89f60-50d5-4e2d-8444-147d39c965e0\") "
Nov 04 01:29:37.816766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.816681    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca" pod "49a89f60-50d5-4e2d-8444-147d39c965e0" (UID: "49a89f60-50d5-4e2d-8444-147d39c965e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:29:37.816917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.816893    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:37.818479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.818460    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca" pod "49a89f60-50d5-4e2d-8444-147d39c965e0" (UID: "49a89f60-50d5-4e2d-8444-147d39c965e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:29:37.818681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.818662    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w" pod "49a89f60-50d5-4e2d-8444-147d39c965e0" (UID: "49a89f60-50d5-4e2d-8444-147d39c965e0"). InnerVolumeSpecName "kube-api-access-qn74w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:37.917737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.917660    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a89f60-50d5-4e2d-8444-147d39c965e0-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:37.917737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:37.917694    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn74w\" (UniqueName: \"kubernetes.io/projected/49a89f60-50d5-4e2d-8444-147d39c965e0-kube-api-access-qn74w\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:38.382119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:38.382081    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts" event={"ID":"49a89f60-50d5-4e2d-8444-147d39c965e0","Type":"ContainerDied","Data":"6275bc069a582e0d1d5127137cd8f027c77aadbd279ca6959f252d2bccab3d8b"}
Nov 04 01:29:38.382119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:38.382113    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"
Nov 04 01:29:38.382360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:38.382144    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6275bc069a582e0d1d5127137cd8f027c77aadbd279ca6959f252d2bccab3d8b"
Nov 04 01:29:38.394324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:38.394293    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:39.091575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:39.091541    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:39.778449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:39.778406    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:40.625707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:40.625669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:41.564797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:41.564760    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:41.804680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:41.804555    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:42.086468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:42.086422    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:42.345734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:42.345654    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:42.539022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:42.537434    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" containerID="cri-o://32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315" gracePeriod=45
Nov 04 01:29:42.598534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:42.598336    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:42.804806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:42.798593    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused"
Nov 04 01:29:42.845033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:42.831856    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:43.102318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.102289    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:29:43.255679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.255645    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm\") pod \"c958dc33-8c5d-41ae-8156-41ef575354cf\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") "
Nov 04 01:29:43.255884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.255701    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir\") pod \"c958dc33-8c5d-41ae-8156-41ef575354cf\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") "
Nov 04 01:29:43.255884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.255753    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx\" (UniqueName: \"kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx\") pod \"c958dc33-8c5d-41ae-8156-41ef575354cf\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") "
Nov 04 01:29:43.255884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.255778    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home\") pod \"c958dc33-8c5d-41ae-8156-41ef575354cf\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") "
Nov 04 01:29:43.255884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.255847    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs\") pod \"c958dc33-8c5d-41ae-8156-41ef575354cf\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") "
Nov 04 01:29:43.256161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.255962    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache\") pod \"c958dc33-8c5d-41ae-8156-41ef575354cf\" (UID: \"c958dc33-8c5d-41ae-8156-41ef575354cf\") "
Nov 04 01:29:43.256161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.256143    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir" pod "c958dc33-8c5d-41ae-8156-41ef575354cf" (UID: "c958dc33-8c5d-41ae-8156-41ef575354cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:43.256521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.256436    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache" pod "c958dc33-8c5d-41ae-8156-41ef575354cf" (UID: "c958dc33-8c5d-41ae-8156-41ef575354cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:43.256521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.256458    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home" pod "c958dc33-8c5d-41ae-8156-41ef575354cf" (UID: "c958dc33-8c5d-41ae-8156-41ef575354cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:43.256674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.256602    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:43.256674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.256621    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:43.256674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.256634    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:43.258318 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.258289    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm" pod "c958dc33-8c5d-41ae-8156-41ef575354cf" (UID: "c958dc33-8c5d-41ae-8156-41ef575354cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:29:43.258399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.258355    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx" pod "c958dc33-8c5d-41ae-8156-41ef575354cf" (UID: "c958dc33-8c5d-41ae-8156-41ef575354cf"). InnerVolumeSpecName "kube-api-access-tt4cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:29:43.258853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.258836    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs" pod "c958dc33-8c5d-41ae-8156-41ef575354cf" (UID: "c958dc33-8c5d-41ae-8156-41ef575354cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:29:43.357871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.357839    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4cx\" (UniqueName: \"kubernetes.io/projected/c958dc33-8c5d-41ae-8156-41ef575354cf-kube-api-access-tt4cx\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:43.357871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.357867    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c958dc33-8c5d-41ae-8156-41ef575354cf-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:43.357871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.357879    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c958dc33-8c5d-41ae-8156-41ef575354cf-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:29:43.462975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.462933    5797 generic.go:356] "Generic (PLEG): container finished" podID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerID="32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315" exitCode=0
Nov 04 01:29:43.463233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.463184    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" event={"ID":"c958dc33-8c5d-41ae-8156-41ef575354cf","Type":"ContainerDied","Data":"32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315"}
Nov 04 01:29:43.463314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.463233    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc" event={"ID":"c958dc33-8c5d-41ae-8156-41ef575354cf","Type":"ContainerDied","Data":"4427f30405f28275eb2ab5ba2eb809c74732930b6b1ebc4f8623233ce4e87c5f"}
Nov 04 01:29:43.463314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.463259    5797 scope.go:122] "RemoveContainer" containerID="32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315"
Nov 04 01:29:43.463496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.463430    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"
Nov 04 01:29:43.477378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.477353    5797 scope.go:122] "RemoveContainer" containerID="32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315"
Nov 04 01:29:43.477701 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:29:43.477679    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315\": container with ID starting with 32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315 not found: ID does not exist" containerID="32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315"
Nov 04 01:29:43.477802 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.477709    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315"} err="failed to get container status \"32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315\": rpc error: code = NotFound desc = could not find container \"32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315\": container with ID starting with 32a82356f47a5ebe4d5c230d5252adc303167fef4523776c7c069ab14b955315 not found: ID does not exist"
Nov 04 01:29:43.495012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.494728    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"]
Nov 04 01:29:43.498883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.498859    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a5115a/premium-w3-a5115a-kserve-6d9d6c5c69-gv7xc"]
Nov 04 01:29:43.803086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:43.803053    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:44.469769 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:44.469718    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:44.477523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:44.477494    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" path="/var/lib/kubelet/pods/c958dc33-8c5d-41ae-8156-41ef575354cf/volumes"
Nov 04 01:29:45.195320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:45.195288    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:45.676441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:45.676405    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:45.881289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:45.881251    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:46.097927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:46.097847    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:46.732938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:46.732904    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:46.951505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:46.951472    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:47.152691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:47.152660    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:47.803857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:47.803811    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:48.026540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.026505    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:48.071587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.071505    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"]
Nov 04 01:29:48.072185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.072165    5797 cpu_manager.go:414] "containerMap: removing container" podUID="49a89f60-50d5-4e2d-8444-147d39c965e0" containerName="revoke-keys"
Nov 04 01:29:48.072240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.072187    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a89f60-50d5-4e2d-8444-147d39c965e0" containerName="revoke-keys"
Nov 04 01:29:48.072240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.072214    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main"
Nov 04 01:29:48.072240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.072219    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main"
Nov 04 01:29:48.072343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.072302    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c958dc33-8c5d-41ae-8156-41ef575354cf" containerName="main"
Nov 04 01:29:48.072343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.072317    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a89f60-50d5-4e2d-8444-147d39c965e0" containerName="revoke-keys"
Nov 04 01:29:48.077301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.077276    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.216047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.157170    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8974fa-openshift-default-dockercfg-h4tg2\""
Nov 04 01:29:48.216047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.177796    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"]
Nov 04 01:29:48.245581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.234443    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:48.260300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260260    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5fc\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260321    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260468 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260425    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260478    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260604    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260675    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260705    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260818    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.260903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.260868    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364272    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364328    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364384    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364434    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364528    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5fc\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364581    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364614    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364675    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.364913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.364725    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.365263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.365237    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.369554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.369529    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.369978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.369958    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.370410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.370390    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.371431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.371405    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.371603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.371478    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.373571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.373545    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.385546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.385520    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.385652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.385555    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5fc\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.547548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.547516    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:48.681888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.681854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:48.843026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.842983    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"]
Nov 04 01:29:48.844539 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:29:48.844510    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32671d4_4e44_413b_812a_8a90c4b80c0d.slice/crio-dfa430dec12d14595b3d5494687ebfc19fc4beb261fba05e1a3237593fe79947 WatchSource:0}: Error finding container dfa430dec12d14595b3d5494687ebfc19fc4beb261fba05e1a3237593fe79947: Status 404 returned error can't find the container with id dfa430dec12d14595b3d5494687ebfc19fc4beb261fba05e1a3237593fe79947
Nov 04 01:29:48.847125 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.847089    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:29:48.847231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.847173    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:29:48.847231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.847219    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:29:48.884186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:48.884160    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:49.506833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.506793    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" event={"ID":"f32671d4-4e44-413b-812a-8a90c4b80c0d","Type":"ContainerStarted","Data":"441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6"}
Nov 04 01:29:49.507036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.506858    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" event={"ID":"f32671d4-4e44-413b-812a-8a90c4b80c0d","Type":"ContainerStarted","Data":"dfa430dec12d14595b3d5494687ebfc19fc4beb261fba05e1a3237593fe79947"}
Nov 04 01:29:49.535388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.535336    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podStartSLOduration=1.535319769 podStartE2EDuration="1.535319769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:49.534113428 +0000 UTC m=+1577.176019706" watchObservedRunningTime="2026-11-04 01:29:49.535319769 +0000 UTC m=+1577.177226089"
Nov 04 01:29:49.574027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.573745    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:49.575679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.575640    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 04 01:29:49.575861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.575709    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 04 01:29:49.700119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:49.700090    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:50.379393 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:50.379352    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:50.552014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:50.550497    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 04 01:29:50.552014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:50.550586    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 04 01:29:50.601657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:50.601618    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:50.917507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:50.917475    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:51.116689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:51.116655    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:51.450778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:51.450731    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:51.558013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:51.553352    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 04 01:29:51.558013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:51.553423    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 04 01:29:51.689869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:51.689833    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:51.930373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:51.930335    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:52.269096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:52.269008    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:52.555917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:52.555828    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 04 01:29:52.556532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:52.556503    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 04 01:29:52.639916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:52.639877    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:53.475843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:53.475804    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:53.567013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:53.563466    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 04 01:29:53.567013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:53.563524    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 04 01:29:53.773354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:53.773249    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:54.420013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:54.413489    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:54.571014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:54.569832    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 04 01:29:54.571014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:54.569928    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 04 01:29:54.796970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:54.796881    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:55.194891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:55.193504    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:55.392668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:55.392632    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:55.553087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:55.553000    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:55.553482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:55.553461    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:55.554113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:55.554091    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:29:55.622414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:55.622330    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:56.074473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:56.074410    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:56.259027 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:29:56.258959    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:29:56.894493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:56.894455    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:57.851123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:57.851054    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:58.060049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:58.060015    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:58.274514 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:58.274474    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:58.688723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:58.688691    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:29:59.392415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:29:59.392386    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:00.089152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.089111    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:00.140267 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.140233    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"]
Nov 04 01:30:00.144655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.144630    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"]
Nov 04 01:30:00.146208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.146190    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.148373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.148351    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"]
Nov 04 01:30:00.148499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.148480    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.155717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.155695    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"]
Nov 04 01:30:00.156528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.155837    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.158923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.158880    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:30:00.165760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.165735    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"]
Nov 04 01:30:00.165896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.165784    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"]
Nov 04 01:30:00.165896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.165813    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"]
Nov 04 01:30:00.165896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.165861    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.166940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.166919    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"]
Nov 04 01:30:00.204126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204098    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.204276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204133    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49th\" (UniqueName: \"kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.204276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204181    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"]
Nov 04 01:30:00.204276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204227    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"]
Nov 04 01:30:00.204276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204232    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.204276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204274    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.204527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204307    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.204527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204317    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwhf\" (UniqueName: \"kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.204527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204453    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzrq\" (UniqueName: \"kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.204685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204578    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhqv\" (UniqueName: \"kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv\") pod \"maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.204685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.204649    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.215297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.212166    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"]
Nov 04 01:30:00.215297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.214225    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.222386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.222362    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"]
Nov 04 01:30:00.222681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.222660    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"]
Nov 04 01:30:00.222832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.222605    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.222832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.222697    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"]
Nov 04 01:30:00.305470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305442    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzrq\" (UniqueName: \"kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.305608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305501    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhqv\" (UniqueName: \"kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv\") pod \"maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.305608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305549    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.305608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305581    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.305748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305607    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-g5tgb\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.305808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305781    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.305846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305830    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49th\" (UniqueName: \"kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.305893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305877    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz88\" (UniqueName: \"kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.305946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305937    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.306051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.305981    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkp88\" (UniqueName: \"kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88\") pod \"maas-api-key-cleanup-29895930-g5tgb\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.306130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306073    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.306197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306150    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzk8\" (UniqueName: \"kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8\") pod \"maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.306267 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306200    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.306330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306275    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwhf\" (UniqueName: \"kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.306439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306417    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.306574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306553    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.306658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306641    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.306759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.306742    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.319581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.319549    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhqv\" (UniqueName: \"kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv\") pod \"maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.319713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.319676    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49th\" (UniqueName: \"kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.319902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.319877    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwhf\" (UniqueName: \"kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.320971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.320950    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzrq\" (UniqueName: \"kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.407829 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.407796    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.408293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.407836    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-g5tgb\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.408293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.407960    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz88\" (UniqueName: \"kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.408293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.408096    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkp88\" (UniqueName: \"kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88\") pod \"maas-api-key-cleanup-29895930-g5tgb\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.408293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.408127    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.408293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.408155    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzk8\" (UniqueName: \"kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8\") pod \"maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.408600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.408580    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-g5tgb\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.408641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.408587    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.408760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.408742    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.421065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.421037    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz88\" (UniqueName: \"kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88\") pod \"maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.421172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.421110    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzk8\" (UniqueName: \"kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8\") pod \"maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.421370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.421349    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkp88\" (UniqueName: \"kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88\") pod \"maas-api-key-cleanup-29895930-g5tgb\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.468077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.468051    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:00.510909 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.510878    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:00.525253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.525228    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:00.536495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.536468    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:00.550642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.550445    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:00.557459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.557394    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:00.567619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.567221    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:00.775228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.775202    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"]
Nov 04 01:30:00.836975 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:00.836041    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf95dba7_ad2b_47da_81cd_aabf52933850.slice/crio-c32846b7ecf051d5c0d3877bd2fc63212c181b5068ac2f54c3fab8d6d8f33ffc WatchSource:0}: Error finding container c32846b7ecf051d5c0d3877bd2fc63212c181b5068ac2f54c3fab8d6d8f33ffc: Status 404 returned error can't find the container with id c32846b7ecf051d5c0d3877bd2fc63212c181b5068ac2f54c3fab8d6d8f33ffc
Nov 04 01:30:00.837180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.837070    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"]
Nov 04 01:30:00.854718 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.854694    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"]
Nov 04 01:30:00.856623 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:00.856601    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf313642e_dc84_4dc9_88c4_945207a5f139.slice/crio-ac7367d317fb17e46c1bf47047a6a43e5d604e18bee35903133f7df0f00ff31d WatchSource:0}: Error finding container ac7367d317fb17e46c1bf47047a6a43e5d604e18bee35903133f7df0f00ff31d: Status 404 returned error can't find the container with id ac7367d317fb17e46c1bf47047a6a43e5d604e18bee35903133f7df0f00ff31d
Nov 04 01:30:00.890778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.890751    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"]
Nov 04 01:30:00.891274 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:00.891246    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff232e0_218b_4f9d_ac06_bfe13f370d8f.slice/crio-db55d59dc99b36ddf25a264b956c07bf4403d6e9a85087f0a21574781a40bd55 WatchSource:0}: Error finding container db55d59dc99b36ddf25a264b956c07bf4403d6e9a85087f0a21574781a40bd55: Status 404 returned error can't find the container with id db55d59dc99b36ddf25a264b956c07bf4403d6e9a85087f0a21574781a40bd55
Nov 04 01:30:00.944066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.944038    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"]
Nov 04 01:30:00.946227 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:00.946203    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d97fda_1f96_4854_ba17_a86a6626e837.slice/crio-2a0b176ca4c05ee9a17ab2e0d571c1c02abff446445289f00fe6cf72c5d369ab WatchSource:0}: Error finding container 2a0b176ca4c05ee9a17ab2e0d571c1c02abff446445289f00fe6cf72c5d369ab: Status 404 returned error can't find the container with id 2a0b176ca4c05ee9a17ab2e0d571c1c02abff446445289f00fe6cf72c5d369ab
Nov 04 01:30:00.955260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.955230    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"]
Nov 04 01:30:00.974662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:00.970208    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"]
Nov 04 01:30:01.045422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.045381    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:01.054881 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:01.054847    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22c3df1_231e_479f_bc29_28c91eeac8b9.slice/crio-b3e2822c01fb0ad37014e399e1382d1257c3c420a7a1dfb93a7e19d953d7bbc1 WatchSource:0}: Error finding container b3e2822c01fb0ad37014e399e1382d1257c3c420a7a1dfb93a7e19d953d7bbc1: Status 404 returned error can't find the container with id b3e2822c01fb0ad37014e399e1382d1257c3c420a7a1dfb93a7e19d953d7bbc1
Nov 04 01:30:01.699529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.699493    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerStarted","Data":"a12ee062cf03681fcb5f99ecdb08bacf293a02f65d9828c94dabf90eff35aea3"}
Nov 04 01:30:01.700010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.699548    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerStarted","Data":"db55d59dc99b36ddf25a264b956c07bf4403d6e9a85087f0a21574781a40bd55"}
Nov 04 01:30:01.701174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.701151    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerStarted","Data":"aca00bcf85747bcd823a2fff17f552ae876dc646bdbbe653993b982f280dca5c"}
Nov 04 01:30:01.701295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.701196    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerStarted","Data":"614f869ec1bd59970499d50ffb99c8284706becf71d9f407961a539d1a48344d"}
Nov 04 01:30:01.702727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.702700    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerStarted","Data":"4c7bdbcac2b7eb20986418a975c508984c0a02a9ee900fe62cd94b06a357c834"}
Nov 04 01:30:01.702849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.702741    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerStarted","Data":"2a0b176ca4c05ee9a17ab2e0d571c1c02abff446445289f00fe6cf72c5d369ab"}
Nov 04 01:30:01.704351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.704320    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerStarted","Data":"3a84476b91ece41e7e2da8e1f2e95aa4177cd20574b6b5a7d80ab6a29d77828c"}
Nov 04 01:30:01.704458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.704356    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerStarted","Data":"d79cc2af00b0cb18e51cc9b62ff5904dd11eee5f6ac9a26bcaa6d9eedec33539"}
Nov 04 01:30:01.705880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.705856    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerStarted","Data":"dab70dcaa957dc9efd5ae1b6f49c77c7de7368bcee2522f37b29e155b7688574"}
Nov 04 01:30:01.706032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.705898    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerStarted","Data":"ac7367d317fb17e46c1bf47047a6a43e5d604e18bee35903133f7df0f00ff31d"}
Nov 04 01:30:01.707446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.707424    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerStarted","Data":"bf93ccc2d6a7345d884b0e1130f1bf312aa962f6a0abb545c4c326601414a969"}
Nov 04 01:30:01.707559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.707467    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerStarted","Data":"c32846b7ecf051d5c0d3877bd2fc63212c181b5068ac2f54c3fab8d6d8f33ffc"}
Nov 04 01:30:01.708895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.708868    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerStarted","Data":"f2033589debae721a94fe59c26fe8fa6f87f084df588d4e7da87d28c9f7f1c92"}
Nov 04 01:30:01.708982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.708906    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerStarted","Data":"b3e2822c01fb0ad37014e399e1382d1257c3c420a7a1dfb93a7e19d953d7bbc1"}
Nov 04 01:30:01.718732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.718693    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" podStartSLOduration=1.718681416 podStartE2EDuration="1.718681416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.716270169 +0000 UTC m=+1589.358176419" watchObservedRunningTime="2026-11-04 01:30:01.718681416 +0000 UTC m=+1589.360587667"
Nov 04 01:30:01.735678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.735635    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" podStartSLOduration=1.735623245 podStartE2EDuration="1.735623245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.733534114 +0000 UTC m=+1589.375440366" watchObservedRunningTime="2026-11-04 01:30:01.735623245 +0000 UTC m=+1589.377529495"
Nov 04 01:30:01.752145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.751612    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" podStartSLOduration=1.7515963220000001 podStartE2EDuration="1.751596322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.750005884 +0000 UTC m=+1589.391912157" watchObservedRunningTime="2026-11-04 01:30:01.751596322 +0000 UTC m=+1589.393502573"
Nov 04 01:30:01.779281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.779233    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" podStartSLOduration=1.779218746 podStartE2EDuration="1.779218746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.7771945 +0000 UTC m=+1589.419100797" watchObservedRunningTime="2026-11-04 01:30:01.779218746 +0000 UTC m=+1589.421125037"
Nov 04 01:30:01.800462 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.800402    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" podStartSLOduration=1.800383466 podStartE2EDuration="1.800383466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.797481824 +0000 UTC m=+1589.439388074" watchObservedRunningTime="2026-11-04 01:30:01.800383466 +0000 UTC m=+1589.442289717"
Nov 04 01:30:01.817983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.817850    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" podStartSLOduration=1.817833523 podStartE2EDuration="1.817833523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.814741494 +0000 UTC m=+1589.456647769" watchObservedRunningTime="2026-11-04 01:30:01.817833523 +0000 UTC m=+1589.459739775"
Nov 04 01:30:01.836019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.835945    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" podStartSLOduration=1.835931556 podStartE2EDuration="1.835931556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:01.834769915 +0000 UTC m=+1589.476676159" watchObservedRunningTime="2026-11-04 01:30:01.835931556 +0000 UTC m=+1589.477837805"
Nov 04 01:30:01.976395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:01.976362    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:02.372972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.372925    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:02.755965 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.739654    5797 generic.go:356] "Generic (PLEG): container finished" podID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerID="3a84476b91ece41e7e2da8e1f2e95aa4177cd20574b6b5a7d80ab6a29d77828c" exitCode=60
Nov 04 01:30:02.755965 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.739758    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerDied","Data":"3a84476b91ece41e7e2da8e1f2e95aa4177cd20574b6b5a7d80ab6a29d77828c"}
Nov 04 01:30:02.755965 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.740346    5797 scope.go:122] "RemoveContainer" containerID="3a84476b91ece41e7e2da8e1f2e95aa4177cd20574b6b5a7d80ab6a29d77828c"
Nov 04 01:30:02.765098 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.765066    5797 generic.go:356] "Generic (PLEG): container finished" podID="f313642e-dc84-4dc9-88c4-945207a5f139" containerID="dab70dcaa957dc9efd5ae1b6f49c77c7de7368bcee2522f37b29e155b7688574" exitCode=60
Nov 04 01:30:02.765248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.765160    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerDied","Data":"dab70dcaa957dc9efd5ae1b6f49c77c7de7368bcee2522f37b29e155b7688574"}
Nov 04 01:30:02.765617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.765599    5797 scope.go:122] "RemoveContainer" containerID="dab70dcaa957dc9efd5ae1b6f49c77c7de7368bcee2522f37b29e155b7688574"
Nov 04 01:30:02.768446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.768421    5797 generic.go:356] "Generic (PLEG): container finished" podID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerID="bf93ccc2d6a7345d884b0e1130f1bf312aa962f6a0abb545c4c326601414a969" exitCode=60
Nov 04 01:30:02.768568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.768551    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerDied","Data":"bf93ccc2d6a7345d884b0e1130f1bf312aa962f6a0abb545c4c326601414a969"}
Nov 04 01:30:02.768933 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.768898    5797 scope.go:122] "RemoveContainer" containerID="bf93ccc2d6a7345d884b0e1130f1bf312aa962f6a0abb545c4c326601414a969"
Nov 04 01:30:02.773438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.773413    5797 generic.go:356] "Generic (PLEG): container finished" podID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerID="f2033589debae721a94fe59c26fe8fa6f87f084df588d4e7da87d28c9f7f1c92" exitCode=60
Nov 04 01:30:02.774111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.774093    5797 scope.go:122] "RemoveContainer" containerID="f2033589debae721a94fe59c26fe8fa6f87f084df588d4e7da87d28c9f7f1c92"
Nov 04 01:30:02.775026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.774407    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerDied","Data":"f2033589debae721a94fe59c26fe8fa6f87f084df588d4e7da87d28c9f7f1c92"}
Nov 04 01:30:02.776895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.776870    5797 generic.go:356] "Generic (PLEG): container finished" podID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerID="a12ee062cf03681fcb5f99ecdb08bacf293a02f65d9828c94dabf90eff35aea3" exitCode=60
Nov 04 01:30:02.777094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.776959    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerDied","Data":"a12ee062cf03681fcb5f99ecdb08bacf293a02f65d9828c94dabf90eff35aea3"}
Nov 04 01:30:02.777211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.777197    5797 scope.go:122] "RemoveContainer" containerID="a12ee062cf03681fcb5f99ecdb08bacf293a02f65d9828c94dabf90eff35aea3"
Nov 04 01:30:02.779234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.779215    5797 generic.go:356] "Generic (PLEG): container finished" podID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerID="aca00bcf85747bcd823a2fff17f552ae876dc646bdbbe653993b982f280dca5c" exitCode=60
Nov 04 01:30:02.779332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.779300    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerDied","Data":"aca00bcf85747bcd823a2fff17f552ae876dc646bdbbe653993b982f280dca5c"}
Nov 04 01:30:02.779932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.779482    5797 scope.go:122] "RemoveContainer" containerID="aca00bcf85747bcd823a2fff17f552ae876dc646bdbbe653993b982f280dca5c"
Nov 04 01:30:02.781180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.781158    5797 generic.go:356] "Generic (PLEG): container finished" podID="36d97fda-1f96-4854-ba17-a86a6626e837" containerID="4c7bdbcac2b7eb20986418a975c508984c0a02a9ee900fe62cd94b06a357c834" exitCode=7
Nov 04 01:30:02.781287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.781229    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerDied","Data":"4c7bdbcac2b7eb20986418a975c508984c0a02a9ee900fe62cd94b06a357c834"}
Nov 04 01:30:02.781464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.781447    5797 scope.go:122] "RemoveContainer" containerID="4c7bdbcac2b7eb20986418a975c508984c0a02a9ee900fe62cd94b06a357c834"
Nov 04 01:30:02.794549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:02.791325    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:03.599558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.599486    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:03.791061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.791025    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerStarted","Data":"3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2"}
Nov 04 01:30:03.793143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.793120    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerStarted","Data":"747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65"}
Nov 04 01:30:03.795082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.795061    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerStarted","Data":"415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095"}
Nov 04 01:30:03.797138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.797106    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerStarted","Data":"bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2"}
Nov 04 01:30:03.805071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.802405    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerStarted","Data":"8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7"}
Nov 04 01:30:03.805564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.805535    5797 generic.go:356] "Generic (PLEG): container finished" podID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerID="2935e2862285fe7f927f8e2d060e072fa9dc0af6da02c78d03cb88dfeb5ae92a" exitCode=60
Nov 04 01:30:03.805680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.805641    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerDied","Data":"2935e2862285fe7f927f8e2d060e072fa9dc0af6da02c78d03cb88dfeb5ae92a"}
Nov 04 01:30:03.805789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.805694    5797 scope.go:122] "RemoveContainer" containerID="aca00bcf85747bcd823a2fff17f552ae876dc646bdbbe653993b982f280dca5c"
Nov 04 01:30:03.809373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.806178    5797 scope.go:122] "RemoveContainer" containerID="2935e2862285fe7f927f8e2d060e072fa9dc0af6da02c78d03cb88dfeb5ae92a"
Nov 04 01:30:03.809373 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:03.806534    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4_odh-ai-gateway-infra(913fe31c-d62b-4117-bf9c-2a5908942cd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4"
Nov 04 01:30:03.853465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.843587    5797 generic.go:356] "Generic (PLEG): container finished" podID="36d97fda-1f96-4854-ba17-a86a6626e837" containerID="f1332570cd6fec2959e7d6087aca68098897a70aac0337737e39619644a8aade" exitCode=7
Nov 04 01:30:03.853465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.843705    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerDied","Data":"f1332570cd6fec2959e7d6087aca68098897a70aac0337737e39619644a8aade"}
Nov 04 01:30:03.853465 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.844227    5797 scope.go:122] "RemoveContainer" containerID="f1332570cd6fec2959e7d6087aca68098897a70aac0337737e39619644a8aade"
Nov 04 01:30:03.853465 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:03.844617    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp_odh-ai-gateway-infra(36d97fda-1f96-4854-ba17-a86a6626e837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" podUID="36d97fda-1f96-4854-ba17-a86a6626e837"
Nov 04 01:30:03.876413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:03.876381    5797 scope.go:122] "RemoveContainer" containerID="4c7bdbcac2b7eb20986418a975c508984c0a02a9ee900fe62cd94b06a357c834"
Nov 04 01:30:04.479728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.479690    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:04.861805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.861724    5797 generic.go:356] "Generic (PLEG): container finished" podID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerID="bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2" exitCode=60
Nov 04 01:30:04.862288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.861803    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerDied","Data":"bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2"}
Nov 04 01:30:04.862288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.861872    5797 scope.go:122] "RemoveContainer" containerID="f2033589debae721a94fe59c26fe8fa6f87f084df588d4e7da87d28c9f7f1c92"
Nov 04 01:30:04.862288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.862237    5797 scope.go:122] "RemoveContainer" containerID="bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2"
Nov 04 01:30:04.862613 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.862586    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-g5tgb_odh-ai-gateway-infra(e22c3df1-231e-479f-bc29-28c91eeac8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9"
Nov 04 01:30:04.864172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.864149    5797 generic.go:356] "Generic (PLEG): container finished" podID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerID="8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7" exitCode=60
Nov 04 01:30:04.864293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.864231    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerDied","Data":"8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7"}
Nov 04 01:30:04.864473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.864458    5797 scope.go:122] "RemoveContainer" containerID="8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7"
Nov 04 01:30:04.864776 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.864755    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz_odh-ai-gateway-infra(2ff232e0-218b-4f9d-ac06-bfe13f370d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f"
Nov 04 01:30:04.866316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.866294    5797 scope.go:122] "RemoveContainer" containerID="2935e2862285fe7f927f8e2d060e072fa9dc0af6da02c78d03cb88dfeb5ae92a"
Nov 04 01:30:04.866591 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.866569    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4_odh-ai-gateway-infra(913fe31c-d62b-4117-bf9c-2a5908942cd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4"
Nov 04 01:30:04.867904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.867884    5797 scope.go:122] "RemoveContainer" containerID="f1332570cd6fec2959e7d6087aca68098897a70aac0337737e39619644a8aade"
Nov 04 01:30:04.868245 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.868220    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp_odh-ai-gateway-infra(36d97fda-1f96-4854-ba17-a86a6626e837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" podUID="36d97fda-1f96-4854-ba17-a86a6626e837"
Nov 04 01:30:04.869916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.869892    5797 generic.go:356] "Generic (PLEG): container finished" podID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerID="3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2" exitCode=60
Nov 04 01:30:04.870055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.869970    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerDied","Data":"3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2"}
Nov 04 01:30:04.870186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.870169    5797 scope.go:122] "RemoveContainer" containerID="3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2"
Nov 04 01:30:04.870890 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.870470    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s_odh-ai-gateway-infra(e51cf802-8608-4f7a-8d72-0063f3f7d0e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2"
Nov 04 01:30:04.872471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.872454    5797 generic.go:356] "Generic (PLEG): container finished" podID="f313642e-dc84-4dc9-88c4-945207a5f139" containerID="747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65" exitCode=60
Nov 04 01:30:04.872556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.872511    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerDied","Data":"747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65"}
Nov 04 01:30:04.872766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.872749    5797 scope.go:122] "RemoveContainer" containerID="747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65"
Nov 04 01:30:04.873040 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.873017    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z_odh-ai-gateway-infra(f313642e-dc84-4dc9-88c4-945207a5f139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" podUID="f313642e-dc84-4dc9-88c4-945207a5f139"
Nov 04 01:30:04.874454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.874433    5797 generic.go:356] "Generic (PLEG): container finished" podID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerID="415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095" exitCode=60
Nov 04 01:30:04.874530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.874457    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerDied","Data":"415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095"}
Nov 04 01:30:04.874710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.874694    5797 scope.go:122] "RemoveContainer" containerID="415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095"
Nov 04 01:30:04.874949 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:04.874927    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc_odh-ai-gateway-infra(bf95dba7-ad2b-47da-81cd-aabf52933850)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850"
Nov 04 01:30:04.876326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.876309    5797 scope.go:122] "RemoveContainer" containerID="a12ee062cf03681fcb5f99ecdb08bacf293a02f65d9828c94dabf90eff35aea3"
Nov 04 01:30:04.890556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.890535    5797 scope.go:122] "RemoveContainer" containerID="3a84476b91ece41e7e2da8e1f2e95aa4177cd20574b6b5a7d80ab6a29d77828c"
Nov 04 01:30:04.904816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.904799    5797 scope.go:122] "RemoveContainer" containerID="dab70dcaa957dc9efd5ae1b6f49c77c7de7368bcee2522f37b29e155b7688574"
Nov 04 01:30:04.918093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:04.918072    5797 scope.go:122] "RemoveContainer" containerID="bf93ccc2d6a7345d884b0e1130f1bf312aa962f6a0abb545c4c326601414a969"
Nov 04 01:30:05.463107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:05.463073    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:05.884341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:05.884304    5797 scope.go:122] "RemoveContainer" containerID="3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2"
Nov 04 01:30:05.884853 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:05.884669    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s_odh-ai-gateway-infra(e51cf802-8608-4f7a-8d72-0063f3f7d0e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2"
Nov 04 01:30:05.886302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:05.886280    5797 scope.go:122] "RemoveContainer" containerID="747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65"
Nov 04 01:30:05.886591 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:05.886567    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z_odh-ai-gateway-infra(f313642e-dc84-4dc9-88c4-945207a5f139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" podUID="f313642e-dc84-4dc9-88c4-945207a5f139"
Nov 04 01:30:05.888291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:05.888269    5797 scope.go:122] "RemoveContainer" containerID="415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095"
Nov 04 01:30:05.888554 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:05.888529    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc_odh-ai-gateway-infra(bf95dba7-ad2b-47da-81cd-aabf52933850)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850"
Nov 04 01:30:05.889977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:05.889957    5797 scope.go:122] "RemoveContainer" containerID="bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2"
Nov 04 01:30:05.890240 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:05.890212    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-g5tgb_odh-ai-gateway-infra(e22c3df1-231e-479f-bc29-28c91eeac8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9"
Nov 04 01:30:05.891826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:05.891810    5797 scope.go:122] "RemoveContainer" containerID="8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7"
Nov 04 01:30:05.892041 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:05.891985    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz_odh-ai-gateway-infra(2ff232e0-218b-4f9d-ac06-bfe13f370d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f"
Nov 04 01:30:06.080704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:06.080660    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:06.283879 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:06.283791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:06.490016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:06.489267    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:07.039012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.038838    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:07.562471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.562438    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:07.774068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.768656    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"]
Nov 04 01:30:07.774068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.768914    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" containerID="cri-o://34f78ede5b29d54d9e73ce2985933050be0ddf4ce2203f914af4ab65b9176108" gracePeriod=30
Nov 04 01:30:07.774068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.772709    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:07.774971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.774896    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"]
Nov 04 01:30:07.777059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.775237    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" containerID="cri-o://9d12511a8edc6d3f7fc2dc6a8207d513b8269f051d0af858b373540f3192108d" gracePeriod=30
Nov 04 01:30:07.996869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:07.996778    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:08.794445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:08.794411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:09.483437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:09.483401    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:30:09.483603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:09.483458    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:30:09.490235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:09.490205    5797 patch_prober.go:29] interesting pod/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:30:09.490370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:09.490252    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:30:09.668490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:09.668456    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:10.429455 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:10.429422    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:11.190871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:11.190840    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:11.874810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:11.874766    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:12.681224 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:12.681189    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:12.948783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:12.948710    5797 generic.go:356] "Generic (PLEG): container finished" podID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerID="34f78ede5b29d54d9e73ce2985933050be0ddf4ce2203f914af4ab65b9176108" exitCode=0
Nov 04 01:30:12.949255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:12.948793    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" event={"ID":"a00f3b41-2394-44ed-9802-cb14cb74f785","Type":"ContainerDied","Data":"34f78ede5b29d54d9e73ce2985933050be0ddf4ce2203f914af4ab65b9176108"}
Nov 04 01:30:12.955356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:12.955332    5797 generic.go:356] "Generic (PLEG): container finished" podID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerID="9d12511a8edc6d3f7fc2dc6a8207d513b8269f051d0af858b373540f3192108d" exitCode=0
Nov 04 01:30:12.955470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:12.955368    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" event={"ID":"94f234fa-0b09-4589-a6ab-0f57217e890a","Type":"ContainerDied","Data":"9d12511a8edc6d3f7fc2dc6a8207d513b8269f051d0af858b373540f3192108d"}
Nov 04 01:30:13.048002 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.047960    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:30:13.095845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.095810    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:30:13.154094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154058    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154105    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154131    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154158    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154233    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154275    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154374    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154426    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154455    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154582    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154620    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154654    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154697    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.154857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154769    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154803    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.154857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154841    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.155191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154879    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs\") pod \"a00f3b41-2394-44ed-9802-cb14cb74f785\" (UID: \"a00f3b41-2394-44ed-9802-cb14cb74f785\") "
Nov 04 01:30:13.155191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154919    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs\") pod \"94f234fa-0b09-4589-a6ab-0f57217e890a\" (UID: \"94f234fa-0b09-4589-a6ab-0f57217e890a\") "
Nov 04 01:30:13.155191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.154956    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.155569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.155545    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.155667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.155594    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.155862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.155838    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.155944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.155876    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.155944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.155894    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.156189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.156118    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.156273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.156202    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.156273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.156233    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.156273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.156240    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:13.156620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.156598    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:13.157377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.157352    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.157780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.157759    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "kube-api-access-jbmd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:13.158146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.158112    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:30:13.158146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.158119    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:13.158298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.158144    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:30:13.158357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.158305    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:13.158600 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.158584    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo" pod "94f234fa-0b09-4589-a6ab-0f57217e890a" (UID: "94f234fa-0b09-4589-a6ab-0f57217e890a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:30:13.159701 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.159678    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj" pod "a00f3b41-2394-44ed-9802-cb14cb74f785" (UID: "a00f3b41-2394-44ed-9802-cb14cb74f785"). InnerVolumeSpecName "kube-api-access-nvwpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256523    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvwpj\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-kube-api-access-nvwpj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256556    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256569    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f234fa-0b09-4589-a6ab-0f57217e890a-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256577    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256586    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256595    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256604    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256612    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f3b41-2394-44ed-9802-cb14cb74f785-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256621    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256629    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256637    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256645    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256653    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256661    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmd5\" (UniqueName: \"kubernetes.io/projected/94f234fa-0b09-4589-a6ab-0f57217e890a-kube-api-access-jbmd5\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256669    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256677    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f3b41-2394-44ed-9802-cb14cb74f785-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.256977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.256686    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f234fa-0b09-4589-a6ab-0f57217e890a-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:13.480088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.480051    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:13.963519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.963487    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"
Nov 04 01:30:13.964027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.963491    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd" event={"ID":"a00f3b41-2394-44ed-9802-cb14cb74f785","Type":"ContainerDied","Data":"3525a18fc005113172ce90a69ab98aead6976fbd225eeb91046d4c5f3410276f"}
Nov 04 01:30:13.964027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.963622    5797 scope.go:122] "RemoveContainer" containerID="34f78ede5b29d54d9e73ce2985933050be0ddf4ce2203f914af4ab65b9176108"
Nov 04 01:30:13.965164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.965138    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"
Nov 04 01:30:13.965322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.965163    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6" event={"ID":"94f234fa-0b09-4589-a6ab-0f57217e890a","Type":"ContainerDied","Data":"bbe721d10f2aeab7d237c1a5f317b81d89593e3263eae367473fde15783cff30"}
Nov 04 01:30:13.978790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.978769    5797 scope.go:122] "RemoveContainer" containerID="9d12511a8edc6d3f7fc2dc6a8207d513b8269f051d0af858b373540f3192108d"
Nov 04 01:30:13.992280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.992259    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"]
Nov 04 01:30:13.997722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:13.997699    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-fkkpd"]
Nov 04 01:30:14.012641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.012608    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"]
Nov 04 01:30:14.019456 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.019427    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a5115a-openshift-default-59f76958f9-gk7j6"]
Nov 04 01:30:14.157027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.156969    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:14.471525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.471437    5797 scope.go:122] "RemoveContainer" containerID="2935e2862285fe7f927f8e2d060e072fa9dc0af6da02c78d03cb88dfeb5ae92a"
Nov 04 01:30:14.479056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.479031    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" path="/var/lib/kubelet/pods/94f234fa-0b09-4589-a6ab-0f57217e890a/volumes"
Nov 04 01:30:14.479539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.479525    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" path="/var/lib/kubelet/pods/a00f3b41-2394-44ed-9802-cb14cb74f785/volumes"
Nov 04 01:30:14.583463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.583429    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:14.907380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:14.907343    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:15.006332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:15.006243    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerStarted","Data":"1bce132ef4f567dead9a01a7a6509c0be71e66b666630126baaafbcc5b499051"}
Nov 04 01:30:15.125285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:15.125246    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:15.354288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:15.354097    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:16.059119 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.056747    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"]
Nov 04 01:30:16.063140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.063100    5797 generic.go:356] "Generic (PLEG): container finished" podID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerID="1bce132ef4f567dead9a01a7a6509c0be71e66b666630126baaafbcc5b499051" exitCode=60
Nov 04 01:30:16.066013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.063859    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerDied","Data":"1bce132ef4f567dead9a01a7a6509c0be71e66b666630126baaafbcc5b499051"}
Nov 04 01:30:16.066564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.066520    5797 scope.go:122] "RemoveContainer" containerID="2935e2862285fe7f927f8e2d060e072fa9dc0af6da02c78d03cb88dfeb5ae92a"
Nov 04 01:30:16.185104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.176604    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:16.418732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.418698    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:16.476916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.476875    5797 scope.go:122] "RemoveContainer" containerID="bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2"
Nov 04 01:30:16.563036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.563002    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88\" (UniqueName: \"kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88\") pod \"913fe31c-d62b-4117-bf9c-2a5908942cd4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") "
Nov 04 01:30:16.563164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.563071    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca\") pod \"913fe31c-d62b-4117-bf9c-2a5908942cd4\" (UID: \"913fe31c-d62b-4117-bf9c-2a5908942cd4\") "
Nov 04 01:30:16.563471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.563442    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca" pod "913fe31c-d62b-4117-bf9c-2a5908942cd4" (UID: "913fe31c-d62b-4117-bf9c-2a5908942cd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:16.563606 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.563592    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913fe31c-d62b-4117-bf9c-2a5908942cd4-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:16.564894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.564872    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88" pod "913fe31c-d62b-4117-bf9c-2a5908942cd4" (UID: "913fe31c-d62b-4117-bf9c-2a5908942cd4"). InnerVolumeSpecName "kube-api-access-hcz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:16.664911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.664871    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcz88\" (UniqueName: \"kubernetes.io/projected/913fe31c-d62b-4117-bf9c-2a5908942cd4-kube-api-access-hcz88\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:16.889139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:16.889110    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:17.074441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.074359    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"
Nov 04 01:30:17.074441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.074400    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4" event={"ID":"913fe31c-d62b-4117-bf9c-2a5908942cd4","Type":"ContainerDied","Data":"614f869ec1bd59970499d50ffb99c8284706becf71d9f407961a539d1a48344d"}
Nov 04 01:30:17.075008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.074459    5797 scope.go:122] "RemoveContainer" containerID="1bce132ef4f567dead9a01a7a6509c0be71e66b666630126baaafbcc5b499051"
Nov 04 01:30:17.076615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.076585    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerStarted","Data":"8589805c64d694015b0455a472c497a3704a27eab71728ea53172072d150615a"}
Nov 04 01:30:17.115533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.115502    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"]
Nov 04 01:30:17.120070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.120044    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1036e8-29895930-k7bb4"]
Nov 04 01:30:17.170284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.170248    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:17.471320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.471288    5797 scope.go:122] "RemoveContainer" containerID="8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7"
Nov 04 01:30:17.471524 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.471499    5797 scope.go:122] "RemoveContainer" containerID="747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65"
Nov 04 01:30:17.471623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.471602    5797 scope.go:122] "RemoveContainer" containerID="3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2"
Nov 04 01:30:17.471711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.471679    5797 scope.go:122] "RemoveContainer" containerID="415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095"
Nov 04 01:30:17.472394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.471966    5797 scope.go:122] "RemoveContainer" containerID="f1332570cd6fec2959e7d6087aca68098897a70aac0337737e39619644a8aade"
Nov 04 01:30:17.783154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:17.783128    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:18.085432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.085393    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerStarted","Data":"30150da4558a06a2f7d987b9ab3b490d73c078d11793996357c9d138b0b801a3"}
Nov 04 01:30:18.088070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.088037    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerStarted","Data":"6831d85549a413a115028a7d37994e2ad27fa2714e9596a2b8d576bf631ce5e5"}
Nov 04 01:30:18.090168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.090141    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerStarted","Data":"ba374aa173bc00377fadaa68f8ce0320e354d6fd3413bba308544bbe7f57cd51"}
Nov 04 01:30:18.091974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.091950    5797 generic.go:356] "Generic (PLEG): container finished" podID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerID="8589805c64d694015b0455a472c497a3704a27eab71728ea53172072d150615a" exitCode=60
Nov 04 01:30:18.092105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.092017    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerDied","Data":"8589805c64d694015b0455a472c497a3704a27eab71728ea53172072d150615a"}
Nov 04 01:30:18.092105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.092082    5797 scope.go:122] "RemoveContainer" containerID="bb63ae07fe3b6ce71a66fb237b2506a425d058ec053c6b195e69483ad01916c2"
Nov 04 01:30:18.092369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.092352    5797 scope.go:122] "RemoveContainer" containerID="8589805c64d694015b0455a472c497a3704a27eab71728ea53172072d150615a"
Nov 04 01:30:18.092585 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:18.092565    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-g5tgb_odh-ai-gateway-infra(e22c3df1-231e-479f-bc29-28c91eeac8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9"
Nov 04 01:30:18.093918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.093900    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerStarted","Data":"824a1f337063718d785fdcf8e1a0464627047eac1b16a5e9880a78a5c3af891f"}
Nov 04 01:30:18.097476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.097446    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerStarted","Data":"45f62d20bab7042cef66aeeeb7e1773dbeaeaaf5af1d79699feb7c291b8d923c"}
Nov 04 01:30:18.099979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.099954    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"]
Nov 04 01:30:18.100604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100586    5797 cpu_manager.go:414] "containerMap: removing container" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:18.100604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100604    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:18.100752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100620    5797 cpu_manager.go:414] "containerMap: removing container" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy"
Nov 04 01:30:18.100752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100626    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy"
Nov 04 01:30:18.100752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100643    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy"
Nov 04 01:30:18.100752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100649    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy"
Nov 04 01:30:18.100752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100674    5797 cpu_manager.go:414] "containerMap: removing container" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:18.100752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100679    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:18.101121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100765    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:18.101121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100777    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f234fa-0b09-4589-a6ab-0f57217e890a" containerName="istio-proxy"
Nov 04 01:30:18.101121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100787    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00f3b41-2394-44ed-9802-cb14cb74f785" containerName="istio-proxy"
Nov 04 01:30:18.101121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.100794    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:18.104496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.104479    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.114386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.114363    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"]
Nov 04 01:30:18.120121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.120096    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"]
Nov 04 01:30:18.285215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.285181    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.285410 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.285344    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.285558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.285534    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.285665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.285607    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.285918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.285882    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdsc\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.286020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.285936    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.286091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.286070    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.286146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.286115    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.286201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.286189    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387380    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387457    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387514    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdsc\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387543    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387599    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387638    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387687    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.387801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387744    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.388081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387819    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.388081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387852    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.388081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.387920    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.388081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.388020    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.388284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.388245    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.388552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.388524    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.390226 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.390204    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.390495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.390460    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.400156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.400125    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.400294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.400129    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdsc\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc\") pod \"e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") " pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.424186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.424157    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:18.479294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.479257    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" path="/var/lib/kubelet/pods/913fe31c-d62b-4117-bf9c-2a5908942cd4/volumes"
Nov 04 01:30:18.678460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.678427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:18.714495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.714467    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"]
Nov 04 01:30:18.753469 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:18.753441    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c1e230_165b_43d7_8e13_e04412657a83.slice/crio-360912fbafe300e0db7919b7845b08ed3ae9f03b6cb3bfd01a078288e06275bc WatchSource:0}: Error finding container 360912fbafe300e0db7919b7845b08ed3ae9f03b6cb3bfd01a078288e06275bc: Status 404 returned error can't find the container with id 360912fbafe300e0db7919b7845b08ed3ae9f03b6cb3bfd01a078288e06275bc
Nov 04 01:30:18.756159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.756125    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:30:18.756277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.756194    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:30:18.756277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:18.756225    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:30:19.109585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.109490    5797 generic.go:356] "Generic (PLEG): container finished" podID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerID="824a1f337063718d785fdcf8e1a0464627047eac1b16a5e9880a78a5c3af891f" exitCode=60
Nov 04 01:30:19.110101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.109578    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerDied","Data":"824a1f337063718d785fdcf8e1a0464627047eac1b16a5e9880a78a5c3af891f"}
Nov 04 01:30:19.110101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.109687    5797 scope.go:122] "RemoveContainer" containerID="8ccd38215b7c615c22e2df53bf974dd06931eafd8dc09fb4307ce3f76b0727f7"
Nov 04 01:30:19.110287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.110240    5797 scope.go:122] "RemoveContainer" containerID="824a1f337063718d785fdcf8e1a0464627047eac1b16a5e9880a78a5c3af891f"
Nov 04 01:30:19.110604 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:19.110579    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz_odh-ai-gateway-infra(2ff232e0-218b-4f9d-ac06-bfe13f370d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f"
Nov 04 01:30:19.111872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.111849    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" event={"ID":"22c1e230-165b-43d7-8e13-e04412657a83","Type":"ContainerStarted","Data":"371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43"}
Nov 04 01:30:19.111927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.111900    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" event={"ID":"22c1e230-165b-43d7-8e13-e04412657a83","Type":"ContainerStarted","Data":"360912fbafe300e0db7919b7845b08ed3ae9f03b6cb3bfd01a078288e06275bc"}
Nov 04 01:30:19.117555 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.117531    5797 generic.go:356] "Generic (PLEG): container finished" podID="36d97fda-1f96-4854-ba17-a86a6626e837" containerID="45f62d20bab7042cef66aeeeb7e1773dbeaeaaf5af1d79699feb7c291b8d923c" exitCode=60
Nov 04 01:30:19.117675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.117593    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerDied","Data":"45f62d20bab7042cef66aeeeb7e1773dbeaeaaf5af1d79699feb7c291b8d923c"}
Nov 04 01:30:19.117949 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.117928    5797 scope.go:122] "RemoveContainer" containerID="45f62d20bab7042cef66aeeeb7e1773dbeaeaaf5af1d79699feb7c291b8d923c"
Nov 04 01:30:19.119174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.118187    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"]
Nov 04 01:30:19.119174 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:19.118289    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp_odh-ai-gateway-infra(36d97fda-1f96-4854-ba17-a86a6626e837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" podUID="36d97fda-1f96-4854-ba17-a86a6626e837"
Nov 04 01:30:19.122684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.122660    5797 generic.go:356] "Generic (PLEG): container finished" podID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerID="30150da4558a06a2f7d987b9ab3b490d73c078d11793996357c9d138b0b801a3" exitCode=60
Nov 04 01:30:19.122800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.122744    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerDied","Data":"30150da4558a06a2f7d987b9ab3b490d73c078d11793996357c9d138b0b801a3"}
Nov 04 01:30:19.125188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.125163    5797 generic.go:356] "Generic (PLEG): container finished" podID="f313642e-dc84-4dc9-88c4-945207a5f139" containerID="6831d85549a413a115028a7d37994e2ad27fa2714e9596a2b8d576bf631ce5e5" exitCode=60
Nov 04 01:30:19.125288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.125242    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerDied","Data":"6831d85549a413a115028a7d37994e2ad27fa2714e9596a2b8d576bf631ce5e5"}
Nov 04 01:30:19.125607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.125589    5797 scope.go:122] "RemoveContainer" containerID="6831d85549a413a115028a7d37994e2ad27fa2714e9596a2b8d576bf631ce5e5"
Nov 04 01:30:19.126070 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:19.125953    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z_odh-ai-gateway-infra(f313642e-dc84-4dc9-88c4-945207a5f139)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" podUID="f313642e-dc84-4dc9-88c4-945207a5f139"
Nov 04 01:30:19.129818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.128471    5797 generic.go:356] "Generic (PLEG): container finished" podID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerID="ba374aa173bc00377fadaa68f8ce0320e354d6fd3413bba308544bbe7f57cd51" exitCode=60
Nov 04 01:30:19.131445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.131242    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerDied","Data":"ba374aa173bc00377fadaa68f8ce0320e354d6fd3413bba308544bbe7f57cd51"}
Nov 04 01:30:19.135016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.132607    5797 scope.go:122] "RemoveContainer" containerID="ba374aa173bc00377fadaa68f8ce0320e354d6fd3413bba308544bbe7f57cd51"
Nov 04 01:30:19.135016 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:19.132937    5797 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc_odh-ai-gateway-infra(bf95dba7-ad2b-47da-81cd-aabf52933850)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850"
Nov 04 01:30:19.135016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.133025    5797 scope.go:122] "RemoveContainer" containerID="f1332570cd6fec2959e7d6087aca68098897a70aac0337737e39619644a8aade"
Nov 04 01:30:19.157867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.157772    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"]
Nov 04 01:30:19.163787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.163486    5797 scope.go:122] "RemoveContainer" containerID="3a32924e1943b24dbdb3da02a1dea1e7b51c1da7bad951424aef2b203be578c2"
Nov 04 01:30:19.200682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.195312    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"]
Nov 04 01:30:19.206444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.201366    5797 scope.go:122] "RemoveContainer" containerID="747a73ce96c4b8c0d8ea657b24cf3b4548546db5ddfa7972326d7426259c8e65"
Nov 04 01:30:19.210978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.210929    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podStartSLOduration=1.210914812 podStartE2EDuration="1.210914812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:19.201746371 +0000 UTC m=+1606.843652642" watchObservedRunningTime="2026-11-04 01:30:19.210914812 +0000 UTC m=+1606.852821065"
Nov 04 01:30:19.226196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.226091    5797 scope.go:122] "RemoveContainer" containerID="415f62010ba6e5ed1f969e4d4e685b1f94a14881a799623ee7b13b230462c095"
Nov 04 01:30:19.228156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.226382    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"]
Nov 04 01:30:19.244093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.243967    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"]
Nov 04 01:30:19.328204 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.328174    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:19.405173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.405140    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:19.405828 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.405777    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca\") pod \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") "
Nov 04 01:30:19.406857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.406075    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th\" (UniqueName: \"kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th\") pod \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\" (UID: \"e51cf802-8608-4f7a-8d72-0063f3f7d0e2\") "
Nov 04 01:30:19.407276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.407252    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca" pod "e51cf802-8608-4f7a-8d72-0063f3f7d0e2" (UID: "e51cf802-8608-4f7a-8d72-0063f3f7d0e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:19.409324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.409282    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th" pod "e51cf802-8608-4f7a-8d72-0063f3f7d0e2" (UID: "e51cf802-8608-4f7a-8d72-0063f3f7d0e2"). InnerVolumeSpecName "kube-api-access-d49th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:19.426719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.425248    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:19.428150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.428119    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 01:30:19.428289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.428178    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 01:30:19.527827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.509506    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca\") pod \"e22c3df1-231e-479f-bc29-28c91eeac8b9\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") "
Nov 04 01:30:19.528254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.528222    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88\" (UniqueName: \"kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88\") pod \"e22c3df1-231e-479f-bc29-28c91eeac8b9\" (UID: \"e22c3df1-231e-479f-bc29-28c91eeac8b9\") "
Nov 04 01:30:19.540047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.540008    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49th\" (UniqueName: \"kubernetes.io/projected/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-kube-api-access-d49th\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:19.547449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.547410    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51cf802-8608-4f7a-8d72-0063f3f7d0e2-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:19.547748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.522837    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca" pod "e22c3df1-231e-479f-bc29-28c91eeac8b9" (UID: "e22c3df1-231e-479f-bc29-28c91eeac8b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:19.569049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.568691    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:19.574415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.574376    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88" pod "e22c3df1-231e-479f-bc29-28c91eeac8b9" (UID: "e22c3df1-231e-479f-bc29-28c91eeac8b9"). InnerVolumeSpecName "kube-api-access-dkp88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:19.656173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.656064    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkp88\" (UniqueName: \"kubernetes.io/projected/e22c3df1-231e-479f-bc29-28c91eeac8b9-kube-api-access-dkp88\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:19.656173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:19.656110    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22c3df1-231e-479f-bc29-28c91eeac8b9-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.180615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.180579    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s" event={"ID":"e51cf802-8608-4f7a-8d72-0063f3f7d0e2","Type":"ContainerDied","Data":"d79cc2af00b0cb18e51cc9b62ff5904dd11eee5f6ac9a26bcaa6d9eedec33539"}
Nov 04 01:30:20.180615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.180599    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"
Nov 04 01:30:20.181203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.180644    5797 scope.go:122] "RemoveContainer" containerID="30150da4558a06a2f7d987b9ab3b490d73c078d11793996357c9d138b0b801a3"
Nov 04 01:30:20.186115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.186086    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb" event={"ID":"e22c3df1-231e-479f-bc29-28c91eeac8b9","Type":"ContainerDied","Data":"b3e2822c01fb0ad37014e399e1382d1257c3c420a7a1dfb93a7e19d953d7bbc1"}
Nov 04 01:30:20.186240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.186187    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"
Nov 04 01:30:20.195788 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.195771    5797 scope.go:122] "RemoveContainer" containerID="8589805c64d694015b0455a472c497a3704a27eab71728ea53172072d150615a"
Nov 04 01:30:20.246564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.246526    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"]
Nov 04 01:30:20.276609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.276539    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0741ef-29895930-x2t8s"]
Nov 04 01:30:20.348793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.348754    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"]
Nov 04 01:30:20.389452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.389416    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-g5tgb"]
Nov 04 01:30:20.428100 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.428070    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 01:30:20.428305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.428134    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 01:30:20.433418 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.433396    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:20.516482 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.516452    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" path="/var/lib/kubelet/pods/e22c3df1-231e-479f-bc29-28c91eeac8b9/volumes"
Nov 04 01:30:20.517061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.517042    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" path="/var/lib/kubelet/pods/e51cf802-8608-4f7a-8d72-0063f3f7d0e2/volumes"
Nov 04 01:30:20.535098 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.535070    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:20.621641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.621608    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca\") pod \"f313642e-dc84-4dc9-88c4-945207a5f139\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") "
Nov 04 01:30:20.621964 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.621940    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv\" (UniqueName: \"kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv\") pod \"f313642e-dc84-4dc9-88c4-945207a5f139\" (UID: \"f313642e-dc84-4dc9-88c4-945207a5f139\") "
Nov 04 01:30:20.640121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.636536    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca" pod "f313642e-dc84-4dc9-88c4-945207a5f139" (UID: "f313642e-dc84-4dc9-88c4-945207a5f139"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:20.640121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.638240    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv" pod "f313642e-dc84-4dc9-88c4-945207a5f139" (UID: "f313642e-dc84-4dc9-88c4-945207a5f139"). InnerVolumeSpecName "kube-api-access-jjhqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:20.645844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.645552    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:20.658846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.658465    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:20.692593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.692513    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:20.736732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.734771    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf\" (UniqueName: \"kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf\") pod \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") "
Nov 04 01:30:20.736732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.734849    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca\") pod \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\" (UID: \"2ff232e0-218b-4f9d-ac06-bfe13f370d8f\") "
Nov 04 01:30:20.736732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.735173    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f313642e-dc84-4dc9-88c4-945207a5f139-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.736732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.735185    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhqv\" (UniqueName: \"kubernetes.io/projected/f313642e-dc84-4dc9-88c4-945207a5f139-kube-api-access-jjhqv\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.736732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.735444    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca" pod "2ff232e0-218b-4f9d-ac06-bfe13f370d8f" (UID: "2ff232e0-218b-4f9d-ac06-bfe13f370d8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:20.737519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.737490    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf" pod "2ff232e0-218b-4f9d-ac06-bfe13f370d8f" (UID: "2ff232e0-218b-4f9d-ac06-bfe13f370d8f"). InnerVolumeSpecName "kube-api-access-fqwhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:20.836262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.836224    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8\" (UniqueName: \"kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8\") pod \"36d97fda-1f96-4854-ba17-a86a6626e837\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") "
Nov 04 01:30:20.836463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.836314    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca\") pod \"36d97fda-1f96-4854-ba17-a86a6626e837\" (UID: \"36d97fda-1f96-4854-ba17-a86a6626e837\") "
Nov 04 01:30:20.836463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.836369    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca\") pod \"bf95dba7-ad2b-47da-81cd-aabf52933850\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") "
Nov 04 01:30:20.836591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.836526    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq\" (UniqueName: \"kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq\") pod \"bf95dba7-ad2b-47da-81cd-aabf52933850\" (UID: \"bf95dba7-ad2b-47da-81cd-aabf52933850\") "
Nov 04 01:30:20.836729 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.836703    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca" pod "36d97fda-1f96-4854-ba17-a86a6626e837" (UID: "36d97fda-1f96-4854-ba17-a86a6626e837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:20.836805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.836763    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca" pod "bf95dba7-ad2b-47da-81cd-aabf52933850" (UID: "bf95dba7-ad2b-47da-81cd-aabf52933850"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:30:20.837179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.837153    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d97fda-1f96-4854-ba17-a86a6626e837-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.837287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.837184    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf95dba7-ad2b-47da-81cd-aabf52933850-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.837287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.837199    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqwhf\" (UniqueName: \"kubernetes.io/projected/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-kube-api-access-fqwhf\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.837287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.837214    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff232e0-218b-4f9d-ac06-bfe13f370d8f-maas-api-service-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.838458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.838435    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8" pod "36d97fda-1f96-4854-ba17-a86a6626e837" (UID: "36d97fda-1f96-4854-ba17-a86a6626e837"). InnerVolumeSpecName "kube-api-access-nqzk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:20.838619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.838597    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq" pod "bf95dba7-ad2b-47da-81cd-aabf52933850" (UID: "bf95dba7-ad2b-47da-81cd-aabf52933850"). InnerVolumeSpecName "kube-api-access-hqzrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:30:20.939067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.939027    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqzrq\" (UniqueName: \"kubernetes.io/projected/bf95dba7-ad2b-47da-81cd-aabf52933850-kube-api-access-hqzrq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:20.939067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:20.939062    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqzk8\" (UniqueName: \"kubernetes.io/projected/36d97fda-1f96-4854-ba17-a86a6626e837-kube-api-access-nqzk8\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:30:21.206375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.206336    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z" event={"ID":"f313642e-dc84-4dc9-88c4-945207a5f139","Type":"ContainerDied","Data":"ac7367d317fb17e46c1bf47047a6a43e5d604e18bee35903133f7df0f00ff31d"}
Nov 04 01:30:21.206825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.206353    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"
Nov 04 01:30:21.206825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.206402    5797 scope.go:122] "RemoveContainer" containerID="6831d85549a413a115028a7d37994e2ad27fa2714e9596a2b8d576bf631ce5e5"
Nov 04 01:30:21.208109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.208081    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"
Nov 04 01:30:21.208243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.208117    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc" event={"ID":"bf95dba7-ad2b-47da-81cd-aabf52933850","Type":"ContainerDied","Data":"c32846b7ecf051d5c0d3877bd2fc63212c181b5068ac2f54c3fab8d6d8f33ffc"}
Nov 04 01:30:21.216031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.215070    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"
Nov 04 01:30:21.216031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.215089    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz" event={"ID":"2ff232e0-218b-4f9d-ac06-bfe13f370d8f","Type":"ContainerDied","Data":"db55d59dc99b36ddf25a264b956c07bf4403d6e9a85087f0a21574781a40bd55"}
Nov 04 01:30:21.218336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.218314    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"
Nov 04 01:30:21.218454 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.218340    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp" event={"ID":"36d97fda-1f96-4854-ba17-a86a6626e837","Type":"ContainerDied","Data":"2a0b176ca4c05ee9a17ab2e0d571c1c02abff446445289f00fe6cf72c5d369ab"}
Nov 04 01:30:21.219458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.219395    5797 scope.go:122] "RemoveContainer" containerID="ba374aa173bc00377fadaa68f8ce0320e354d6fd3413bba308544bbe7f57cd51"
Nov 04 01:30:21.245321 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.245298    5797 scope.go:122] "RemoveContainer" containerID="824a1f337063718d785fdcf8e1a0464627047eac1b16a5e9880a78a5c3af891f"
Nov 04 01:30:21.278632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.278597    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"]
Nov 04 01:30:21.282256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.282228    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8974fa-29895930-lqx2z"]
Nov 04 01:30:21.288106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.288084    5797 scope.go:122] "RemoveContainer" containerID="45f62d20bab7042cef66aeeeb7e1773dbeaeaaf5af1d79699feb7c291b8d923c"
Nov 04 01:30:21.291120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.291100    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"]
Nov 04 01:30:21.300041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.300019    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-23ef8f-29895930-bkndc"]
Nov 04 01:30:21.319220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.319195    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"]
Nov 04 01:30:21.327533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.327511    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aefc5-29895930-pq2kp"]
Nov 04 01:30:21.340363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.340338    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"]
Nov 04 01:30:21.346330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.346306    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-35980d-29895930-kkksz"]
Nov 04 01:30:21.424796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.424759    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 01:30:21.425017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.424819    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 01:30:21.590912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:21.590879    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:22.297717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.297676    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:22.425050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.424983    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 01:30:22.425224 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.425066    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 01:30:22.480285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.480248    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" path="/var/lib/kubelet/pods/2ff232e0-218b-4f9d-ac06-bfe13f370d8f/volumes"
Nov 04 01:30:22.480793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.480772    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" path="/var/lib/kubelet/pods/36d97fda-1f96-4854-ba17-a86a6626e837/volumes"
Nov 04 01:30:22.481348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.481330    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" path="/var/lib/kubelet/pods/bf95dba7-ad2b-47da-81cd-aabf52933850/volumes"
Nov 04 01:30:22.482267 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:22.482245    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" path="/var/lib/kubelet/pods/f313642e-dc84-4dc9-88c4-945207a5f139/volumes"
Nov 04 01:30:23.192547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:23.192502    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:23.424522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:23.424486    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 01:30:23.424928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:23.424547    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 01:30:23.912044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:23.911976    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:24.425365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:24.425332    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 01:30:24.425788 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:24.425386    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 01:30:24.815303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:24.815210    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:25.430637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:25.430605    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:25.431187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:25.431046    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:25.432234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:25.431966    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:30:25.589310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:25.589271    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:26.265261 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:26.265221    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:30:26.641391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:26.641341    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:27.516926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:27.516894    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:28.277719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:28.277685    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:28.683227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:28.683191    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:28.922539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:28.922512    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:29.125363 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:29.125331    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:29.778316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:29.778280    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:30.480659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.480590    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:30.572346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572306    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"]
Nov 04 01:30:30.572923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572909    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572925    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572939    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572944    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572954    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572959    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572967    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572972    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572977    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:30:30.572983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.572981    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573023    5797 cpu_manager.go:414] "containerMap: removing container" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573028    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573034    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573038    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573046    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573051    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573057    5797 cpu_manager.go:414] "containerMap: removing container" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573061    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573071    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573075    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573081    5797 cpu_manager.go:414] "containerMap: removing container" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573085    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573094    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573098    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573110    5797 cpu_manager.go:414] "containerMap: removing container" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573114    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573120    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573124    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573132    5797 cpu_manager.go:414] "containerMap: removing container" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573136    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573144    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573151    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573232    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573239    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573245    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573251    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573257    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573263    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573269    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573276    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf95dba7-ad2b-47da-81cd-aabf52933850" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573282    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573287    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573291    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573296    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573303    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.573295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573310    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:30:30.574434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573315    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:30:30.574434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.573321    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="913fe31c-d62b-4117-bf9c-2a5908942cd4" containerName="cleanup"
Nov 04 01:30:30.578585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.578564    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.588312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.586812    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a362b6-openshift-default-dockercfg-7zkzs\""
Nov 04 01:30:30.592445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.591804    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"]
Nov 04 01:30:30.715281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715246    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715294    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715314    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715466    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9qk\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715536    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715556    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715626 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715590    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715671    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.715755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.715694    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.816956    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9qk\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817070    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817095    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817129    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817147    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817174    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817288    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817339    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817420    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817577    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817646    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817766    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.817901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.817835    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.818195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.818175    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.819734 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.819713    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.819834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.819786    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.829270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.829244    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.829565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.829541    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9qk\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk\") pod \"e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") " pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:30.897470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:30.897440    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:31.182584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.182542    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:31.182737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.182607    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"]
Nov 04 01:30:31.183342 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:30:31.183319    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb541bb8_5c6a_4de3_a024_f0817755fbbc.slice/crio-6efc656ebd5fc859107f54adb77e9e1059e8466b46ff7dd28e3f67f9734bc8c7 WatchSource:0}: Error finding container 6efc656ebd5fc859107f54adb77e9e1059e8466b46ff7dd28e3f67f9734bc8c7: Status 404 returned error can't find the container with id 6efc656ebd5fc859107f54adb77e9e1059e8466b46ff7dd28e3f67f9734bc8c7
Nov 04 01:30:31.185921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.185890    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:30:31.186014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.185952    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:30:31.186087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.185982    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:30:31.371531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.371489    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" event={"ID":"cb541bb8-5c6a-4de3-a024-f0817755fbbc","Type":"ContainerStarted","Data":"04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c"}
Nov 04 01:30:31.371742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.371567    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" event={"ID":"cb541bb8-5c6a-4de3-a024-f0817755fbbc","Type":"ContainerStarted","Data":"6efc656ebd5fc859107f54adb77e9e1059e8466b46ff7dd28e3f67f9734bc8c7"}
Nov 04 01:30:31.395715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.395661    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podStartSLOduration=1.395645704 podStartE2EDuration="1.395645704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:31.39409846 +0000 UTC m=+1619.036004711" watchObservedRunningTime="2026-11-04 01:30:31.395645704 +0000 UTC m=+1619.037551953"
Nov 04 01:30:31.398673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.398635    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:31.898179 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.898133    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:31.973783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:31.973743    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:32.168722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:32.168618    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:32.788958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:32.788912    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:32.900205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:32.900168    5797 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 01:30:32.900624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:32.900235    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 01:30:33.605288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:33.605251    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:33.898730 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:33.898695    5797 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 01:30:33.898921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:33.898752    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 01:30:34.331445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:34.331357    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:34.898026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:34.897970    5797 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 01:30:34.898208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:34.898045    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 01:30:35.085596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:35.085528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:35.898439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:35.898402    5797 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 01:30:35.899041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:35.899010    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 01:30:35.910438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:35.910411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:36.727014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:36.726600    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:36.898835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:36.898798    5797 patch_prober.go:29] interesting pod/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 01:30:36.899229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:36.898852    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 01:30:37.501483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:37.501434    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:37.917896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:37.917860    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:37.918803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:37.918317    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:37.919286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:37.919261    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:30:38.186060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:38.185934    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:38.861795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:38.861758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:39.600029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:39.599975    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:40.285674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:40.285628    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:41.072263 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:41.072231    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:41.769399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:41.769353    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:42.848181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:42.848094    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:43.252215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:43.252169    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:44.182074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:44.182034    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:44.994364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:44.994332    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:45.688526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:45.688494    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:46.385306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:46.385262    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:47.181228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:47.181193    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:47.884244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:47.884197    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:48.631020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:48.630969    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:49.276280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:49.276242    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:49.971345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:49.971307    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:50.686741 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:50.686697    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:51.378067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:51.376821    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:52.172868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:52.172836    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:52.831030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:52.830983    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:53.497738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:53.497701    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:54.193349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:54.193320    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:54.904693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:54.904660    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:55.685481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:55.685444    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:56.252345 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:30:56.252309    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:30:56.387434 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:56.387401    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:57.101061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:57.101018    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:57.972003 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:57.971955    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:58.884596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:58.884563    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:30:59.821812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:30:59.821772    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:00.693092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:00.693059    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:01.377834 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:01.377787    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:02.156572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:02.156533    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:02.893289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:02.893254    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:03.575448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:03.575411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:04.277891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:04.277856    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:04.981558 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:04.981528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:05.691911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:05.691879    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:06.396413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:06.396381    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:06.612607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:06.612568    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:07.257266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:07.257230    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:07.976323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:07.976280    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:08.625340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:08.625307    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:09.296857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:09.296827    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:09.985322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:09.985285    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:11.022149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:11.022115    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:11.790628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:11.790581    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:12.589672 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:12.589639    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:13.396390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:13.396323    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:14.080506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:14.080469    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:14.782397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:14.782363    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:15.520466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:15.520432    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:16.301145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:16.301109    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:17.056237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:17.056199    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:17.785545 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:17.785507    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:18.478938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:18.478906    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:19.191824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:19.191791    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:20.013800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:20.013758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:20.776487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:20.776454    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:21.586404 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:21.586371    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:22.504914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:22.504874    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:23.182744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:23.182706    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:23.883714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:23.883676    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:24.590168 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:24.590134    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:25.274301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:25.274263    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:25.937539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:25.937479    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:26.252927 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:31:26.252844    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:31:26.692461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:26.692427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:27.579471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:27.579404    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:28.399423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:28.399389    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:29.220337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:29.220299    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:29.898748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:29.898718    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:30.767778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:30.767748    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:31.252770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:31.252734    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:31.691918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:31.691852    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:32.223270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:32.223235    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:32.425691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:32.425655    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:33.194893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:33.194858    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:33.872224 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:33.872188    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:34.634007 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:34.633960    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:35.381195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:35.381159    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:36.172627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:36.172591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:36.873668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:36.873624    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:37.575637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:37.575528    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:38.290760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:38.290711    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:39.171036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:39.170998    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:39.897911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:39.897854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:40.690272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:40.690230    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:40.908219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:40.908178    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:41.484323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:41.484241    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:42.189899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:42.189844    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:42.894491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:42.894457    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:43.581753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:43.581703    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:44.272938 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:44.272887    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:44.981323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:44.981271    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:45.672506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:45.672471    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:46.481823 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:46.481793    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:47.172648 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:47.172614    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:48.030490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:48.030451    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:48.614768 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:48.614727    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:49.310967 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:49.310937    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:49.976828 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:49.976782    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:50.698476 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.698429    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:50.920873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.920837    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"]
Nov 04 01:31:50.921408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921393    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:31:50.921464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921410    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22c3df1-231e-479f-bc29-28c91eeac8b9" containerName="cleanup"
Nov 04 01:31:50.921464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921448    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:31:50.921464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921453    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51cf802-8608-4f7a-8d72-0063f3f7d0e2" containerName="cleanup"
Nov 04 01:31:50.921464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921461    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:31:50.921605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921466    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:31:50.921605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921546    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f313642e-dc84-4dc9-88c4-945207a5f139" containerName="cleanup"
Nov 04 01:31:50.921605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921555    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff232e0-218b-4f9d-ac06-bfe13f370d8f" containerName="cleanup"
Nov 04 01:31:50.921605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.921569    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d97fda-1f96-4854-ba17-a86a6626e837" containerName="cleanup"
Nov 04 01:31:50.925300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.925282    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:50.929893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.929875    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:31:50.930463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.930447    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:31:50.942383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.942353    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"]
Nov 04 01:31:50.990203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.990118    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:50.990360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.990298    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:50.990423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:50.990398    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv6p\" (UniqueName: \"kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.033805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.033770    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"]
Nov 04 01:31:51.045637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.045607    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d38cfb5-29m57"]
Nov 04 01:31:51.091522 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.091488    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.091691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.091619    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.091740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.091697    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv6p\" (UniqueName: \"kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.092336 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.092313    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.093895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.093875    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.104365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.104345    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv6p\" (UniqueName: \"kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p\") pod \"maas-api-revoke-sub-104ebe0d-2mmn7\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.240049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.240012    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:51.517032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.516938    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:51.533052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:51.533021    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"]
Nov 04 01:31:51.535680 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:31:51.535653    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430d8f65_60f7_48d2_a655_c8ad18d337e9.slice/crio-01e6f91b61aebb42b2aeb0244bd6be388e6d5321d31175d03b589a41cb83ff0c WatchSource:0}: Error finding container 01e6f91b61aebb42b2aeb0244bd6be388e6d5321d31175d03b589a41cb83ff0c: Status 404 returned error can't find the container with id 01e6f91b61aebb42b2aeb0244bd6be388e6d5321d31175d03b589a41cb83ff0c
Nov 04 01:31:52.003627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:52.003585    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7" event={"ID":"430d8f65-60f7-48d2-a655-c8ad18d337e9","Type":"ContainerStarted","Data":"089f1c303d6c4bd8d0c41b6f3377e3c650439f5d6e09e875b3f7818682d86a00"}
Nov 04 01:31:52.004064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:52.003651    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7" event={"ID":"430d8f65-60f7-48d2-a655-c8ad18d337e9","Type":"ContainerStarted","Data":"01e6f91b61aebb42b2aeb0244bd6be388e6d5321d31175d03b589a41cb83ff0c"}
Nov 04 01:31:52.025720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:52.025667    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7" podStartSLOduration=2.025652765 podStartE2EDuration="2.025652765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:52.022205081 +0000 UTC m=+1699.664111347" watchObservedRunningTime="2026-11-04 01:31:52.025652765 +0000 UTC m=+1699.667559045"
Nov 04 01:31:52.297158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:52.297128    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:52.479297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:52.479271    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb409099-6c52-489b-957e-f930401ebc21" path="/var/lib/kubelet/pods/bb409099-6c52-489b-957e-f930401ebc21/volumes"
Nov 04 01:31:52.998105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:52.998066    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:53.011538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:53.011509    5797 generic.go:356] "Generic (PLEG): container finished" podID="430d8f65-60f7-48d2-a655-c8ad18d337e9" containerID="089f1c303d6c4bd8d0c41b6f3377e3c650439f5d6e09e875b3f7818682d86a00" exitCode=0
Nov 04 01:31:53.011916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:53.011597    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7" event={"ID":"430d8f65-60f7-48d2-a655-c8ad18d337e9","Type":"ContainerDied","Data":"089f1c303d6c4bd8d0c41b6f3377e3c650439f5d6e09e875b3f7818682d86a00"}
Nov 04 01:31:53.929219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:53.929187    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:54.183415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.183359    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:54.231481 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.231442    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca\") pod \"430d8f65-60f7-48d2-a655-c8ad18d337e9\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") "
Nov 04 01:31:54.231817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.231793    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca\") pod \"430d8f65-60f7-48d2-a655-c8ad18d337e9\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") "
Nov 04 01:31:54.232031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.232013    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p\" (UniqueName: \"kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p\") pod \"430d8f65-60f7-48d2-a655-c8ad18d337e9\" (UID: \"430d8f65-60f7-48d2-a655-c8ad18d337e9\") "
Nov 04 01:31:54.232178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.232139    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca" pod "430d8f65-60f7-48d2-a655-c8ad18d337e9" (UID: "430d8f65-60f7-48d2-a655-c8ad18d337e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:31:54.232762 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.232735    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:31:54.234685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.234649    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca" pod "430d8f65-60f7-48d2-a655-c8ad18d337e9" (UID: "430d8f65-60f7-48d2-a655-c8ad18d337e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:31:54.235677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.235644    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p" pod "430d8f65-60f7-48d2-a655-c8ad18d337e9" (UID: "430d8f65-60f7-48d2-a655-c8ad18d337e9"). InnerVolumeSpecName "kube-api-access-shv6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:31:54.333653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.333621    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shv6p\" (UniqueName: \"kubernetes.io/projected/430d8f65-60f7-48d2-a655-c8ad18d337e9-kube-api-access-shv6p\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:31:54.333653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.333648    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430d8f65-60f7-48d2-a655-c8ad18d337e9-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:31:54.806783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:54.806699    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:55.027314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:55.027282    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"
Nov 04 01:31:55.027495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:55.027311    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7" event={"ID":"430d8f65-60f7-48d2-a655-c8ad18d337e9","Type":"ContainerDied","Data":"01e6f91b61aebb42b2aeb0244bd6be388e6d5321d31175d03b589a41cb83ff0c"}
Nov 04 01:31:55.027495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:55.027363    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e6f91b61aebb42b2aeb0244bd6be388e6d5321d31175d03b589a41cb83ff0c"
Nov 04 01:31:55.736287 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:55.736254    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:56.253188 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:31:56.253148    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:31:56.524107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:56.524033    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:57.464113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:57.464084    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:58.200413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:58.200382    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:59.086748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:59.086709    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:31:59.907812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:31:59.907744    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:01.042380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:01.042347    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:01.272978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:01.272941    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:01.515022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:01.514978    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:01.809615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:01.809491    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:02.139470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.139436    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"]
Nov 04 01:32:02.141135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.141106    5797 cpu_manager.go:414] "containerMap: removing container" podUID="430d8f65-60f7-48d2-a655-c8ad18d337e9" containerName="revoke-keys"
Nov 04 01:32:02.141135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.141133    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430d8f65-60f7-48d2-a655-c8ad18d337e9" containerName="revoke-keys"
Nov 04 01:32:02.141359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.141313    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="430d8f65-60f7-48d2-a655-c8ad18d337e9" containerName="revoke-keys"
Nov 04 01:32:02.145149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.145126    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.145281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.145176    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:02.153951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.153614    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"]
Nov 04 01:32:02.164058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.157265    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:32:02.164058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.158640    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:32:02.181934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.181879    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.182126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.181949    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfgf\" (UniqueName: \"kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.182126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.181984    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.283804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.283771    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.284009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.283815    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfgf\" (UniqueName: \"kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.284009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.283837    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.284442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.284418    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.286275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.286238    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.302509 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.302485    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfgf\" (UniqueName: \"kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf\") pod \"maas-api-revoke-sub-3a6a6d05-fdm4j\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.473181 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.473099    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:02.757090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:02.757057    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"]
Nov 04 01:32:02.758933 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:32:02.758883    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9fe857_4324_41ab_a08e_aa94fca011d6.slice/crio-ad41c2a0f70ec27eaaa8bb28eb06a391122a2d1f2c75864fb02aa66ee5de0809 WatchSource:0}: Error finding container ad41c2a0f70ec27eaaa8bb28eb06a391122a2d1f2c75864fb02aa66ee5de0809: Status 404 returned error can't find the container with id ad41c2a0f70ec27eaaa8bb28eb06a391122a2d1f2c75864fb02aa66ee5de0809
Nov 04 01:32:03.105832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:03.105722    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j" event={"ID":"eb9fe857-4324-41ab-a08e-aa94fca011d6","Type":"ContainerStarted","Data":"ad41c2a0f70ec27eaaa8bb28eb06a391122a2d1f2c75864fb02aa66ee5de0809"}
Nov 04 01:32:03.141983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:03.141947    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:03.886737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:03.886700    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:04.114000 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:04.113968    5797 generic.go:356] "Generic (PLEG): container finished" podID="eb9fe857-4324-41ab-a08e-aa94fca011d6" containerID="63f31d6e2178bc8148e6701a05d32500cc08315f6621ef0a40fc4d6c44076148" exitCode=0
Nov 04 01:32:04.114186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:04.114054    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j" event={"ID":"eb9fe857-4324-41ab-a08e-aa94fca011d6","Type":"ContainerDied","Data":"63f31d6e2178bc8148e6701a05d32500cc08315f6621ef0a40fc4d6c44076148"}
Nov 04 01:32:04.582930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:04.582826    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:05.274662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.274635    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:05.329082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.327863    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca\") pod \"eb9fe857-4324-41ab-a08e-aa94fca011d6\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") "
Nov 04 01:32:05.329082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.327959    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca\") pod \"eb9fe857-4324-41ab-a08e-aa94fca011d6\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") "
Nov 04 01:32:05.329082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.328108    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf\" (UniqueName: \"kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf\") pod \"eb9fe857-4324-41ab-a08e-aa94fca011d6\" (UID: \"eb9fe857-4324-41ab-a08e-aa94fca011d6\") "
Nov 04 01:32:05.378702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.355218    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca" pod "eb9fe857-4324-41ab-a08e-aa94fca011d6" (UID: "eb9fe857-4324-41ab-a08e-aa94fca011d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:05.378702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.368156    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca" pod "eb9fe857-4324-41ab-a08e-aa94fca011d6" (UID: "eb9fe857-4324-41ab-a08e-aa94fca011d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:32:05.385255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.384700    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf" pod "eb9fe857-4324-41ab-a08e-aa94fca011d6" (UID: "eb9fe857-4324-41ab-a08e-aa94fca011d6"). InnerVolumeSpecName "kube-api-access-dnfgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:05.406780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.406746    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:05.429493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.429466    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:05.429493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.429493    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9fe857-4324-41ab-a08e-aa94fca011d6-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:05.429655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:05.429503    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnfgf\" (UniqueName: \"kubernetes.io/projected/eb9fe857-4324-41ab-a08e-aa94fca011d6-kube-api-access-dnfgf\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:06.160018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:06.152281    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j" event={"ID":"eb9fe857-4324-41ab-a08e-aa94fca011d6","Type":"ContainerDied","Data":"ad41c2a0f70ec27eaaa8bb28eb06a391122a2d1f2c75864fb02aa66ee5de0809"}
Nov 04 01:32:06.160018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:06.152352    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad41c2a0f70ec27eaaa8bb28eb06a391122a2d1f2c75864fb02aa66ee5de0809"
Nov 04 01:32:06.160018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:06.152430    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"
Nov 04 01:32:06.184937 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:06.184902    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:07.043351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:07.043308    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"]
Nov 04 01:32:07.080574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:07.080536    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb279d1d-hw57h"]
Nov 04 01:32:07.135873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:07.135835    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:08.092897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:08.092860    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:08.479298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:08.479257    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db694a80-da92-4cab-85b5-ed1919c17da6" path="/var/lib/kubelet/pods/db694a80-da92-4cab-85b5-ed1919c17da6/volumes"
Nov 04 01:32:08.887823 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:08.887789    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:09.701502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:09.701467    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:10.339320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.339288    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"]
Nov 04 01:32:10.339844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.339829    5797 cpu_manager.go:414] "containerMap: removing container" podUID="eb9fe857-4324-41ab-a08e-aa94fca011d6" containerName="revoke-keys"
Nov 04 01:32:10.339896 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.339845    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9fe857-4324-41ab-a08e-aa94fca011d6" containerName="revoke-keys"
Nov 04 01:32:10.339998 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.339979    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9fe857-4324-41ab-a08e-aa94fca011d6" containerName="revoke-keys"
Nov 04 01:32:10.343815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.343789    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.352638 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.352612    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:32:10.354227 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.354211    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:32:10.359033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.359009    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"]
Nov 04 01:32:10.401023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.400946    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.401250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.401221    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwmx\" (UniqueName: \"kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.401351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.401333    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.508948 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.508909    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.509471 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.509371    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwmx\" (UniqueName: \"kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.509688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.509669    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.511049 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.511014    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.517112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.517085    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.524310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.524285    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwmx\" (UniqueName: \"kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx\") pod \"maas-api-revoke-sub-9e42255f-vb5hh\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.547882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.547855    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:10.660886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.660844    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:10.939903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:10.939876    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"]
Nov 04 01:32:10.941458 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:32:10.941431    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a6b7fc_d80a_4b3e_bab6_3436034c53b0.slice/crio-ee0e209bd341f6528c4f44bec3a9ea3de93aecd9b90e0237bcefe0379423c819 WatchSource:0}: Error finding container ee0e209bd341f6528c4f44bec3a9ea3de93aecd9b90e0237bcefe0379423c819: Status 404 returned error can't find the container with id ee0e209bd341f6528c4f44bec3a9ea3de93aecd9b90e0237bcefe0379423c819
Nov 04 01:32:11.205226 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:11.205129    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh" event={"ID":"77a6b7fc-d80a-4b3e-bab6-3436034c53b0","Type":"ContainerStarted","Data":"ee0e209bd341f6528c4f44bec3a9ea3de93aecd9b90e0237bcefe0379423c819"}
Nov 04 01:32:11.306012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:11.305565    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:12.216413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:12.216372    5797 generic.go:356] "Generic (PLEG): container finished" podID="77a6b7fc-d80a-4b3e-bab6-3436034c53b0" containerID="69ee276e6ac093fed6070e7722745a8859eadc18fd0ee4df2d91a323bf668aa2" exitCode=0
Nov 04 01:32:12.216936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:12.216460    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh" event={"ID":"77a6b7fc-d80a-4b3e-bab6-3436034c53b0","Type":"ContainerDied","Data":"69ee276e6ac093fed6070e7722745a8859eadc18fd0ee4df2d91a323bf668aa2"}
Nov 04 01:32:12.391356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:12.391325    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:12.902397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:12.902358    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:13.141604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.141570    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:13.393118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.393092    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:13.455092 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.455058    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca\") pod \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") "
Nov 04 01:32:13.455295 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.455125    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca\") pod \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") "
Nov 04 01:32:13.455420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.455397    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx\" (UniqueName: \"kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx\") pod \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\" (UID: \"77a6b7fc-d80a-4b3e-bab6-3436034c53b0\") "
Nov 04 01:32:13.455563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.455535    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca" pod "77a6b7fc-d80a-4b3e-bab6-3436034c53b0" (UID: "77a6b7fc-d80a-4b3e-bab6-3436034c53b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:13.455892 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.455870    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:13.457547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.457521    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca" pod "77a6b7fc-d80a-4b3e-bab6-3436034c53b0" (UID: "77a6b7fc-d80a-4b3e-bab6-3436034c53b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:32:13.458025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.457957    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx" pod "77a6b7fc-d80a-4b3e-bab6-3436034c53b0" (UID: "77a6b7fc-d80a-4b3e-bab6-3436034c53b0"). InnerVolumeSpecName "kube-api-access-zcwmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:13.475077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.475022    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:13.557563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.557521    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwmx\" (UniqueName: \"kubernetes.io/projected/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-kube-api-access-zcwmx\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:13.557563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.557564    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a6b7fc-d80a-4b3e-bab6-3436034c53b0-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:13.748447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:13.748358    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:14.231518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:14.231487    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh" event={"ID":"77a6b7fc-d80a-4b3e-bab6-3436034c53b0","Type":"ContainerDied","Data":"ee0e209bd341f6528c4f44bec3a9ea3de93aecd9b90e0237bcefe0379423c819"}
Nov 04 01:32:14.231518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:14.231513    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"
Nov 04 01:32:14.231725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:14.231533    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0e209bd341f6528c4f44bec3a9ea3de93aecd9b90e0237bcefe0379423c819"
Nov 04 01:32:14.672869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:14.672834    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:15.669682 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:15.669650    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:16.576272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:16.576233    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:17.044386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:17.044348    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"]
Nov 04 01:32:17.046950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:17.046923    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ea0dd7-t985w"]
Nov 04 01:32:17.385192 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:17.385151    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:18.179191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:18.179148    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:18.482029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:18.481932    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629ae3d0-ccae-4570-b4da-83dbb480ec5e" path="/var/lib/kubelet/pods/629ae3d0-ccae-4570-b4da-83dbb480ec5e/volumes"
Nov 04 01:32:18.537601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:18.537561    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:18.744536 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:18.744439    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:19.168932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:19.168828    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:19.928999 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:19.928939    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:20.241280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:20.241188    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:20.700548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:20.700514    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:21.201011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.190711    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:21.542074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.538676    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:21.639014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.633058    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"]
Nov 04 01:32:21.639014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.633983    5797 cpu_manager.go:414] "containerMap: removing container" podUID="77a6b7fc-d80a-4b3e-bab6-3436034c53b0" containerName="revoke-keys"
Nov 04 01:32:21.639014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.634024    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a6b7fc-d80a-4b3e-bab6-3436034c53b0" containerName="revoke-keys"
Nov 04 01:32:21.639014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.634182    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a6b7fc-d80a-4b3e-bab6-3436034c53b0" containerName="revoke-keys"
Nov 04 01:32:21.652330 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.652296    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.657745 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.657658    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"]
Nov 04 01:32:21.660425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.660384    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:32:21.660690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.660678    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:32:21.791700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.791667    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.791884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.791818    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.791884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.791862    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6mt\" (UniqueName: \"kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.871297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.871265    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:21.892893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.892866    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.893061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.892973    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.893061 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.893032    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6mt\" (UniqueName: \"kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.893534 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.893510    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.895458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.895434    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.905341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.905319    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6mt\" (UniqueName: \"kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt\") pod \"maas-api-revoke-sub-ff56a408-sl8n2\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:21.976452 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:21.976421    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:22.255176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:22.255147    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"]
Nov 04 01:32:22.256843 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:32:22.256818    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747867c1_0420_4a36_96a8_8f343a128f4b.slice/crio-a9f6291b3d26b1976f9d4f370b098147660ee9d2f4289421e527f48cd6e2ea5d WatchSource:0}: Error finding container a9f6291b3d26b1976f9d4f370b098147660ee9d2f4289421e527f48cd6e2ea5d: Status 404 returned error can't find the container with id a9f6291b3d26b1976f9d4f370b098147660ee9d2f4289421e527f48cd6e2ea5d
Nov 04 01:32:22.317885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:22.317844    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2" event={"ID":"747867c1-0420-4a36-96a8-8f343a128f4b","Type":"ContainerStarted","Data":"a9f6291b3d26b1976f9d4f370b098147660ee9d2f4289421e527f48cd6e2ea5d"}
Nov 04 01:32:22.681894 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:22.681854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:23.325835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:23.325801    5797 generic.go:356] "Generic (PLEG): container finished" podID="747867c1-0420-4a36-96a8-8f343a128f4b" containerID="3dd3f5059549ac964b3ecbd47a9dac90fad78925e0575cfedfef5f0a6d0d6a9b" exitCode=0
Nov 04 01:32:23.326261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:23.325894    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2" event={"ID":"747867c1-0420-4a36-96a8-8f343a128f4b","Type":"ContainerDied","Data":"3dd3f5059549ac964b3ecbd47a9dac90fad78925e0575cfedfef5f0a6d0d6a9b"}
Nov 04 01:32:23.556324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:23.556243    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:24.360595 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.360562    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:24.508173 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.508150    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:24.634873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.634837    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca\") pod \"747867c1-0420-4a36-96a8-8f343a128f4b\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") "
Nov 04 01:32:24.635091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.634902    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca\") pod \"747867c1-0420-4a36-96a8-8f343a128f4b\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") "
Nov 04 01:32:24.635091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.635076    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt\" (UniqueName: \"kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt\") pod \"747867c1-0420-4a36-96a8-8f343a128f4b\" (UID: \"747867c1-0420-4a36-96a8-8f343a128f4b\") "
Nov 04 01:32:24.635304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.635281    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca" pod "747867c1-0420-4a36-96a8-8f343a128f4b" (UID: "747867c1-0420-4a36-96a8-8f343a128f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:24.636245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.636220    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:24.637360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.637293    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca" pod "747867c1-0420-4a36-96a8-8f343a128f4b" (UID: "747867c1-0420-4a36-96a8-8f343a128f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:32:24.637532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.637493    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt" pod "747867c1-0420-4a36-96a8-8f343a128f4b" (UID: "747867c1-0420-4a36-96a8-8f343a128f4b"). InnerVolumeSpecName "kube-api-access-7x6mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:24.737610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.737575    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747867c1-0420-4a36-96a8-8f343a128f4b-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:24.737610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:24.737606    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x6mt\" (UniqueName: \"kubernetes.io/projected/747867c1-0420-4a36-96a8-8f343a128f4b-kube-api-access-7x6mt\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:25.081805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:25.081697    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:25.351727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:25.351640    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"
Nov 04 01:32:25.351727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:25.351659    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2" event={"ID":"747867c1-0420-4a36-96a8-8f343a128f4b","Type":"ContainerDied","Data":"a9f6291b3d26b1976f9d4f370b098147660ee9d2f4289421e527f48cd6e2ea5d"}
Nov 04 01:32:25.351727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:25.351709    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f6291b3d26b1976f9d4f370b098147660ee9d2f4289421e527f48cd6e2ea5d"
Nov 04 01:32:25.814714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:25.814663    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:26.258909 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:32:26.258865    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:32:26.415912 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:26.415832    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:26.622835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:26.622810    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:26.959372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:26.959261    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:27.292083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:27.289110    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"]
Nov 04 01:32:27.292083 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:27.289359    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy" containerID="cri-o://04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c" gracePeriod=30
Nov 04 01:32:27.424500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:27.424449    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:27.719279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:27.719237    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"]
Nov 04 01:32:27.719585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:27.719555    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy" containerID="cri-o://afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719" gracePeriod=30
Nov 04 01:32:27.807930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:27.807854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:28.024731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.024631    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"]
Nov 04 01:32:28.031329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.031298    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"]
Nov 04 01:32:28.034139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.034115    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfed358-xf5rh"]
Nov 04 01:32:28.036635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.036614    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af3d9893-mhlmk"]
Nov 04 01:32:28.227692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.227659    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:28.478308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.478274    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a135510-23e8-4e79-a77c-42391aa008a3" path="/var/lib/kubelet/pods/2a135510-23e8-4e79-a77c-42391aa008a3/volumes"
Nov 04 01:32:28.478721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.478707    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aba32a2-caa6-4015-8aa0-a1ae9c5ad442" path="/var/lib/kubelet/pods/7aba32a2-caa6-4015-8aa0-a1ae9c5ad442/volumes"
Nov 04 01:32:28.684388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:28.684354    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:29.381176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:29.381144    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:29.823248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:29.823161    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:30.033013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:30.027786    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:30.232974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:30.232940    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:30.783254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:30.783220    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:31.286649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:31.286612    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:31.490527 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:31.490495    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:32.287884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.287849    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:32.504010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.498230    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:32.665821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.665789    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:32:32.701357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.701328    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:32.821559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821530    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.821728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821587    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.821728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821644    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.821728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821671    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.821728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821721    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.822075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821746    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.822075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821805    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.822075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821840    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.822075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821847    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:32.822075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821880    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert\") pod \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\" (UID: \"cb541bb8-5c6a-4de3-a024-f0817755fbbc\") "
Nov 04 01:32:32.822075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.821955    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:32.822371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822190    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:32.822371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822204    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:32.822538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822513    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:32.822678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822659    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.822722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822684    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.822722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822700    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.822722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822714    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.822831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.822729    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.823905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.823877    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:32.824033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.823977    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:32:32.824387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.824364    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:32.824387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.824377    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk" pod "cb541bb8-5c6a-4de3-a024-f0817755fbbc" (UID: "cb541bb8-5c6a-4de3-a024-f0817755fbbc"). InnerVolumeSpecName "kube-api-access-bs9qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:32.905105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.905012    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:32.923653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.923625    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.923653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.923649    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.923808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.923661    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb541bb8-5c6a-4de3-a024-f0817755fbbc-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:32.923808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:32.923671    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9qk\" (UniqueName: \"kubernetes.io/projected/cb541bb8-5c6a-4de3-a024-f0817755fbbc-kube-api-access-bs9qk\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.272276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.272243    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:33.393961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.393935    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:32:33.429930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.429896    5797 generic.go:356] "Generic (PLEG): container finished" podID="262bf953-f280-410a-b214-67b2af0cd83c" containerID="afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719" exitCode=0
Nov 04 01:32:33.430135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.429944    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"
Nov 04 01:32:33.430135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.429979    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" event={"ID":"262bf953-f280-410a-b214-67b2af0cd83c","Type":"ContainerDied","Data":"afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719"}
Nov 04 01:32:33.430135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.430057    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9" event={"ID":"262bf953-f280-410a-b214-67b2af0cd83c","Type":"ContainerDied","Data":"100369bdeaef534ab7bb8fe01824b96fc63f494c7492851a06972acf32c7b68a"}
Nov 04 01:32:33.430135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.430088    5797 scope.go:122] "RemoveContainer" containerID="afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719"
Nov 04 01:32:33.431906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.431882    5797 generic.go:356] "Generic (PLEG): container finished" podID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerID="04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c" exitCode=0
Nov 04 01:32:33.432044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.431967    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"
Nov 04 01:32:33.432113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.432058    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" event={"ID":"cb541bb8-5c6a-4de3-a024-f0817755fbbc","Type":"ContainerDied","Data":"04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c"}
Nov 04 01:32:33.432113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.432102    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7" event={"ID":"cb541bb8-5c6a-4de3-a024-f0817755fbbc","Type":"ContainerDied","Data":"6efc656ebd5fc859107f54adb77e9e1059e8466b46ff7dd28e3f67f9734bc8c7"}
Nov 04 01:32:33.444147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.444131    5797 scope.go:122] "RemoveContainer" containerID="afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719"
Nov 04 01:32:33.444487 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:32:33.444457    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719\": container with ID starting with afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719 not found: ID does not exist" containerID="afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719"
Nov 04 01:32:33.444582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.444499    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719"} err="failed to get container status \"afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719\": rpc error: code = NotFound desc = could not find container \"afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719\": container with ID starting with afe54005de5fc27bb3609bb6a48f18ece393925647370126c2b3707912a8f719 not found: ID does not exist"
Nov 04 01:32:33.444582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.444522    5797 scope.go:122] "RemoveContainer" containerID="04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c"
Nov 04 01:32:33.455856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.455832    5797 scope.go:122] "RemoveContainer" containerID="04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c"
Nov 04 01:32:33.456206 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:32:33.456182    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c\": container with ID starting with 04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c not found: ID does not exist" containerID="04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c"
Nov 04 01:32:33.456325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.456211    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c"} err="failed to get container status \"04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c\": rpc error: code = NotFound desc = could not find container \"04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c\": container with ID starting with 04190e0f9c43d679882aba34f29a5bda350c1d50ce01251fb91a47393926a46c not found: ID does not exist"
Nov 04 01:32:33.459751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.459724    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"]
Nov 04 01:32:33.466089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.466065    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a362b6-openshift-default-59f7dc9f55-ppcn7"]
Nov 04 01:32:33.534188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534106    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534177    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534203    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534301    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534325    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534389    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534416    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534525    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.534615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534555    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket\") pod \"262bf953-f280-410a-b214-67b2af0cd83c\" (UID: \"262bf953-f280-410a-b214-67b2af0cd83c\") "
Nov 04 01:32:33.535077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534691    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:33.535077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534765    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:33.535077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534807    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:33.535077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.534977    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:33.535077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.535054    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:33.535343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.535237    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262bf953-f280-410a-b214-67b2af0cd83c-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.535343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.535251    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.535343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.535261    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.535343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.535269    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.535343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.535280    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.536657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.536618    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:33.536864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.536845    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:32:33.536964 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.536949    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:33.537144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.537126    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv" pod "262bf953-f280-410a-b214-67b2af0cd83c" (UID: "262bf953-f280-410a-b214-67b2af0cd83c"). InnerVolumeSpecName "kube-api-access-zzwbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:33.636787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.636759    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.636787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.636785    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262bf953-f280-410a-b214-67b2af0cd83c-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.637025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.636795    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwbv\" (UniqueName: \"kubernetes.io/projected/262bf953-f280-410a-b214-67b2af0cd83c-kube-api-access-zzwbv\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.637025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.636807    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262bf953-f280-410a-b214-67b2af0cd83c-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:33.757777 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.757702    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"]
Nov 04 01:32:33.761976 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.761945    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-pkmj9"]
Nov 04 01:32:33.982298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:33.982261    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:34.478199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:34.478166    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262bf953-f280-410a-b214-67b2af0cd83c" path="/var/lib/kubelet/pods/262bf953-f280-410a-b214-67b2af0cd83c/volumes"
Nov 04 01:32:34.478681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:34.478667    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" path="/var/lib/kubelet/pods/cb541bb8-5c6a-4de3-a024-f0817755fbbc/volumes"
Nov 04 01:32:34.776775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:34.776701    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:34.975094 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:34.975048    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:35.180669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:35.180634    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:35.591824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:35.591745    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:36.300627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:36.300591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:36.502413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:36.502383    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:36.707515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:36.707476    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:37.028533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:37.028441    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"]
Nov 04 01:32:37.031973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:37.031944    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775fb521-fwt6x"]
Nov 04 01:32:37.577269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:37.577234    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:38.477508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:38.477474    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40" path="/var/lib/kubelet/pods/0f2fd8ca-d920-4e1e-bc1a-6f1c35513f40/volumes"
Nov 04 01:32:39.809702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:39.809649    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:40.015026 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:40.014980    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:40.675549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:40.675506    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:41.373907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:41.373858    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:41.571444 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:41.571412    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:41.912353 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:41.912322    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:42.293062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:42.292516    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:42.580048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:42.579927    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:43.120602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:43.120561    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:43.320692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:43.320654    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:43.970475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:43.970421    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:44.665058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:44.665016    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:45.209796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:45.209760    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:45.547064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:45.540857    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:45.741356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:45.741322    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:46.429740 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:46.429709    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:47.034618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:47.034582    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"]
Nov 04 01:32:47.037145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:47.037122    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979dd2f4-rmpkg"]
Nov 04 01:32:47.130240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:47.130209    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:47.805920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:47.805884    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:48.481368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:48.481332    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5738dfe1-c350-42e8-8a60-82e40169ec3c" path="/var/lib/kubelet/pods/5738dfe1-c350-42e8-8a60-82e40169ec3c/volumes"
Nov 04 01:32:48.498355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:48.498327    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:49.207584 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:49.207545    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:49.811200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:49.811151    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:50.009697 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:50.009648    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:50.224448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:50.224413    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:50.884673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:50.884637    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:51.784272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:51.784239    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:52.461039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:52.460964    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:52.671255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:52.671202    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:52.953702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:52.953670    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:53.281531 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.281443    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"]
Nov 04 01:32:53.281724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.281678    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" containerID="cri-o://441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6" gracePeriod=30
Nov 04 01:32:53.284062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.284040    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"]
Nov 04 01:32:53.284235 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.284215    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" containerID="cri-o://371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43" gracePeriod=30
Nov 04 01:32:53.293799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.293765    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:53.496289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.496233    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:53.698190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:53.698139    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:54.140256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:54.140219    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:54.382653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:54.382613    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:55.111535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:55.111487    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:55.434652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:55.434600    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:32:55.435146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:55.434659    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:32:55.555931 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:55.555897    5797 patch_prober.go:29] interesting pod/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:32:55.556163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:55.555965    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:32:55.776254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:55.776176    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:56.251799 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:32:56.251762    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:32:56.505152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:56.505077    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:56.701289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:56.701254    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:56.965918 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:56.965881    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:57.326674 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:57.326582    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:57.524386 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:57.524357    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:58.144820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.144783    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:58.381780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.381750    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:58.554469 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.554432    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:32:58.606822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.606798    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:32:58.640177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640115    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640223    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640278    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640348    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640423    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640465    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640497    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640599    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.640680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.640656    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo\") pod \"f32671d4-4e44-413b-812a-8a90c4b80c0d\" (UID: \"f32671d4-4e44-413b-812a-8a90c4b80c0d\") "
Nov 04 01:32:58.641124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.641097    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.641329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.641303    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.641614 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.641595    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.641724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.641615    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.641917 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.641898    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:58.644591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.644557    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.646065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.645968    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:32:58.646332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.646309    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:58.646423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.646372    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc" pod "f32671d4-4e44-413b-812a-8a90c4b80c0d" (UID: "f32671d4-4e44-413b-812a-8a90c4b80c0d"). InnerVolumeSpecName "kube-api-access-4n5fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:58.649607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.649581    5797 generic.go:356] "Generic (PLEG): container finished" podID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerID="441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6" exitCode=0
Nov 04 01:32:58.649724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.649632    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"
Nov 04 01:32:58.649724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.649663    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" event={"ID":"f32671d4-4e44-413b-812a-8a90c4b80c0d","Type":"ContainerDied","Data":"441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6"}
Nov 04 01:32:58.649843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.649732    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l" event={"ID":"f32671d4-4e44-413b-812a-8a90c4b80c0d","Type":"ContainerDied","Data":"dfa430dec12d14595b3d5494687ebfc19fc4beb261fba05e1a3237593fe79947"}
Nov 04 01:32:58.649843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.649764    5797 scope.go:122] "RemoveContainer" containerID="441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6"
Nov 04 01:32:58.651470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.651444    5797 generic.go:356] "Generic (PLEG): container finished" podID="22c1e230-165b-43d7-8e13-e04412657a83" containerID="371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43" exitCode=0
Nov 04 01:32:58.651596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.651575    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" event={"ID":"22c1e230-165b-43d7-8e13-e04412657a83","Type":"ContainerDied","Data":"371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43"}
Nov 04 01:32:58.651691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.651624    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs" event={"ID":"22c1e230-165b-43d7-8e13-e04412657a83","Type":"ContainerDied","Data":"360912fbafe300e0db7919b7845b08ed3ae9f03b6cb3bfd01a078288e06275bc"}
Nov 04 01:32:58.651752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.651695    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"
Nov 04 01:32:58.662451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.662421    5797 scope.go:122] "RemoveContainer" containerID="441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6"
Nov 04 01:32:58.662746 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:32:58.662727    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6\": container with ID starting with 441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6 not found: ID does not exist" containerID="441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6"
Nov 04 01:32:58.662819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.662755    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6"} err="failed to get container status \"441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6\": rpc error: code = NotFound desc = could not find container \"441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6\": container with ID starting with 441ae4af9958a4adc0269f66badbec36378c5521ffd79b7f8865229eeec2b5f6 not found: ID does not exist"
Nov 04 01:32:58.662819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.662771    5797 scope.go:122] "RemoveContainer" containerID="371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43"
Nov 04 01:32:58.675637 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.675613    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"]
Nov 04 01:32:58.675728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.675715    5797 scope.go:122] "RemoveContainer" containerID="371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43"
Nov 04 01:32:58.675956 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:32:58.675940    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43\": container with ID starting with 371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43 not found: ID does not exist" containerID="371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43"
Nov 04 01:32:58.676024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.675962    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43"} err="failed to get container status \"371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43\": rpc error: code = NotFound desc = could not find container \"371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43\": container with ID starting with 371cd85189930cfbdee8bbaffd47bfc42580cb7f0c707ac3ee5dac710ecc7f43 not found: ID does not exist"
Nov 04 01:32:58.681979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.681956    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-d8v5l"]
Nov 04 01:32:58.742156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742123    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742182    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742255    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742287    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742315    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742347    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742560 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742525    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742601    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742512    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.742679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742571    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.742679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742663    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token\") pod \"22c1e230-165b-43d7-8e13-e04412657a83\" (UID: \"22c1e230-165b-43d7-8e13-e04412657a83\") "
Nov 04 01:32:58.742826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742674    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.742904 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.742847    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:32:58.743103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743074    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.743283 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743267    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743288    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743303    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743317    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743331    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743345    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743370 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743359    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743374    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32671d4-4e44-413b-812a-8a90c4b80c0d-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743397    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743412    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743425    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c1e230-165b-43d7-8e13-e04412657a83-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743439    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743456    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5fc\" (UniqueName: \"kubernetes.io/projected/f32671d4-4e44-413b-812a-8a90c4b80c0d-kube-api-access-4n5fc\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.743653 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.743472    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32671d4-4e44-413b-812a-8a90c4b80c0d-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.745159 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.745140    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "kube-api-access-zfdsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:58.745243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.745159    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:32:58.745243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.745143    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:32:58.745243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.745180    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo" pod "22c1e230-165b-43d7-8e13-e04412657a83" (UID: "22c1e230-165b-43d7-8e13-e04412657a83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:32:58.844436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.844402    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c1e230-165b-43d7-8e13-e04412657a83-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.844436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.844431    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c1e230-165b-43d7-8e13-e04412657a83-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.844436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.844443    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdsc\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-kube-api-access-zfdsc\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.844683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.844454    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c1e230-165b-43d7-8e13-e04412657a83-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:32:58.977552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.977518    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"]
Nov 04 01:32:58.983978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:58.983951    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8974fa-openshift-default-78d69f7b89-nhgqs"]
Nov 04 01:32:59.246120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:59.243660    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:32:59.476654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:32:59.476613    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:00.100852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:00.100815    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:00.477757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:00.477722    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c1e230-165b-43d7-8e13-e04412657a83" path="/var/lib/kubelet/pods/22c1e230-165b-43d7-8e13-e04412657a83/volumes"
Nov 04 01:33:00.478266 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:00.478242    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" path="/var/lib/kubelet/pods/f32671d4-4e44-413b-812a-8a90c4b80c0d/volumes"
Nov 04 01:33:00.789717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:00.789632    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:01.489201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:01.489163    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:01.682438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:01.682402    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:02.290790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:02.290761    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:02.862698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:02.862665    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:03.271144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:03.271111    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:03.549012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:03.548918    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:03.749077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:03.749035    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:04.209413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:04.209364    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:04.874383 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:04.874341    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:05.072731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:05.072697    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:05.585795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:05.585755    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:06.144575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:06.144541    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:06.429027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:06.428913    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:07.101572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:07.101541    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:07.669662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:07.669621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:07.945048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:07.944966    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:08.189680 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:08.189645    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:08.383921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:08.383889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:08.699012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:08.695465    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:08.911766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:08.911733    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:09.487777 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:09.487745    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:09.693449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:09.693411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:10.269125 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:10.269089    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:10.943754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:10.943714    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:11.543088 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:11.543054    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:11.741759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:11.741727    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:12.160479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:12.160402    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:12.523118 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:12.517964    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:12.728863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:12.728831    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:12.764467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:12.764433    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"]
Nov 04 01:33:12.764717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:12.764622    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy" containerID="cri-o://ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91" gracePeriod=30
Nov 04 01:33:12.974629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:12.974591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:13.367647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.367619    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"]
Nov 04 01:33:13.368142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368128    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368143    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368154    5797 cpu_manager.go:414] "containerMap: removing container" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368159    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368176    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368181    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368200    5797 cpu_manager.go:414] "containerMap: removing container" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy"
Nov 04 01:33:13.368203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368205    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368215    5797 cpu_manager.go:414] "containerMap: removing container" podUID="747867c1-0420-4a36-96a8-8f343a128f4b" containerName="revoke-keys"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368219    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747867c1-0420-4a36-96a8-8f343a128f4b" containerName="revoke-keys"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368289    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c1e230-165b-43d7-8e13-e04412657a83" containerName="istio-proxy"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368299    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="747867c1-0420-4a36-96a8-8f343a128f4b" containerName="revoke-keys"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368317    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32671d4-4e44-413b-812a-8a90c4b80c0d" containerName="istio-proxy"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368328    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="262bf953-f280-410a-b214-67b2af0cd83c" containerName="istio-proxy"
Nov 04 01:33:13.368449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.368340    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb541bb8-5c6a-4de3-a024-f0817755fbbc" containerName="istio-proxy"
Nov 04 01:33:13.373499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.373481    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.381766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.381743    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:33:13.382409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.382388    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:33:13.385237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.385208    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"]
Nov 04 01:33:13.409694 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.409639    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:13.457858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.457822    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.458021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.457967    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82w5s\" (UniqueName: \"kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.458102 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.458080    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.558945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.558912    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.559131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.559048    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82w5s\" (UniqueName: \"kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.559131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.559091    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.559603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.559581    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.561463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.561442    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.571801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.571775    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82w5s\" (UniqueName: \"kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s\") pod \"maas-api-revoke-sub-568ccf3d-rn258\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.690316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.690204    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:13.972741 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:13.972715    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"]
Nov 04 01:33:13.975287 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:33:13.975258    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61b0760_a032_45c1_b3d9_b95049996d1f.slice/crio-bee4a497cbd4090cebf3134d3e387655db8a223bd8af76c956e7217554ab63bb WatchSource:0}: Error finding container bee4a497cbd4090cebf3134d3e387655db8a223bd8af76c956e7217554ab63bb: Status 404 returned error can't find the container with id bee4a497cbd4090cebf3134d3e387655db8a223bd8af76c956e7217554ab63bb
Nov 04 01:33:14.196963 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:14.196928    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:14.688936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:14.688871    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:14.836062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:14.835975    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258" event={"ID":"d61b0760-a032-45c1-b3d9-b95049996d1f","Type":"ContainerStarted","Data":"dd82cf5e636e520dfb70d97375842a62fd37ed22c3fe8301b86ac1a36851f6b8"}
Nov 04 01:33:14.836255 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:14.836083    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258" event={"ID":"d61b0760-a032-45c1-b3d9-b95049996d1f","Type":"ContainerStarted","Data":"bee4a497cbd4090cebf3134d3e387655db8a223bd8af76c956e7217554ab63bb"}
Nov 04 01:33:14.866876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:14.866813    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258" podStartSLOduration=1.866794718 podStartE2EDuration="1.866794718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:14.856300771 +0000 UTC m=+1782.498207045" watchObservedRunningTime="2026-11-04 01:33:14.866794718 +0000 UTC m=+1782.508700968"
Nov 04 01:33:14.887660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:14.887632    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:15.178257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:15.178211    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:15.843528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:15.843495    5797 generic.go:356] "Generic (PLEG): container finished" podID="d61b0760-a032-45c1-b3d9-b95049996d1f" containerID="dd82cf5e636e520dfb70d97375842a62fd37ed22c3fe8301b86ac1a36851f6b8" exitCode=0
Nov 04 01:33:15.843928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:15.843544    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258" event={"ID":"d61b0760-a032-45c1-b3d9-b95049996d1f","Type":"ContainerDied","Data":"dd82cf5e636e520dfb70d97375842a62fd37ed22c3fe8301b86ac1a36851f6b8"}
Nov 04 01:33:15.935112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:15.935070    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:16.025540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:16.025498    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"]
Nov 04 01:33:16.027711 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:16.027687    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c58e0a-zjmdk"]
Nov 04 01:33:16.131280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:16.131248    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:16.478573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:16.478494    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f384bd-68ce-4fd5-8da5-8b5197924582" path="/var/lib/kubelet/pods/70f384bd-68ce-4fd5-8da5-8b5197924582/volumes"
Nov 04 01:33:16.789443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:16.789368    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:16.993927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:16.993901    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:17.112172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.112073    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca\") pod \"d61b0760-a032-45c1-b3d9-b95049996d1f\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") "
Nov 04 01:33:17.112365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.112236    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca\") pod \"d61b0760-a032-45c1-b3d9-b95049996d1f\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") "
Nov 04 01:33:17.112365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.112275    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s\" (UniqueName: \"kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s\") pod \"d61b0760-a032-45c1-b3d9-b95049996d1f\" (UID: \"d61b0760-a032-45c1-b3d9-b95049996d1f\") "
Nov 04 01:33:17.112596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.112572    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca" pod "d61b0760-a032-45c1-b3d9-b95049996d1f" (UID: "d61b0760-a032-45c1-b3d9-b95049996d1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:17.112708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.112694    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:17.114340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.114318    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca" pod "d61b0760-a032-45c1-b3d9-b95049996d1f" (UID: "d61b0760-a032-45c1-b3d9-b95049996d1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:17.114445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.114423    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s" pod "d61b0760-a032-45c1-b3d9-b95049996d1f" (UID: "d61b0760-a032-45c1-b3d9-b95049996d1f"). InnerVolumeSpecName "kube-api-access-82w5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:17.214448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.214416    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82w5s\" (UniqueName: \"kubernetes.io/projected/d61b0760-a032-45c1-b3d9-b95049996d1f-kube-api-access-82w5s\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:17.214448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.214446    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61b0760-a032-45c1-b3d9-b95049996d1f-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:17.487177 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.487141    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:17.860925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.860893    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"
Nov 04 01:33:17.860925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.860908    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258" event={"ID":"d61b0760-a032-45c1-b3d9-b95049996d1f","Type":"ContainerDied","Data":"bee4a497cbd4090cebf3134d3e387655db8a223bd8af76c956e7217554ab63bb"}
Nov 04 01:33:17.861234 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.860959    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bee4a497cbd4090cebf3134d3e387655db8a223bd8af76c956e7217554ab63bb"
Nov 04 01:33:17.862566 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:17.862536    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"]
Nov 04 01:33:18.053920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.053884    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:18.067548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.067523    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:33:18.126206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126175    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126226    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126313    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126392    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126469    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126508    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126553    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126564    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:18.126853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126622    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.126853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126629    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:18.126853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.126665    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f\") pod \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\" (UID: \"6b6ded4e-6d49-4cda-bbb9-243a999d05e9\") "
Nov 04 01:33:18.127366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.127169    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.127366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.127225    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.127532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.127366    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:18.127532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.127437    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:18.128160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.128127    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:18.129258 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.129228    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:18.129470 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.129341    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:33:18.129659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.129641    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "kube-api-access-tkc6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:18.129873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.129844    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token" pod "6b6ded4e-6d49-4cda-bbb9-243a999d05e9" (UID: "6b6ded4e-6d49-4cda-bbb9-243a999d05e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.231934    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.231983    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.232018    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.232041    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.232055    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.232068    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.235700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.232081    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkc6f\" (UniqueName: \"kubernetes.io/projected/6b6ded4e-6d49-4cda-bbb9-243a999d05e9-kube-api-access-tkc6f\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:18.271564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.271508    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:18.474387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.474351    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:18.711593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.711546    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:18.870359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.870326    5797 generic.go:356] "Generic (PLEG): container finished" podID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerID="ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91" exitCode=0
Nov 04 01:33:18.870578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.870386    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"
Nov 04 01:33:18.870578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.870426    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" event={"ID":"6b6ded4e-6d49-4cda-bbb9-243a999d05e9","Type":"ContainerDied","Data":"ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91"}
Nov 04 01:33:18.870578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.870479    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj" event={"ID":"6b6ded4e-6d49-4cda-bbb9-243a999d05e9","Type":"ContainerDied","Data":"de4ced1fe9d6ee13a9ccb119e8fbcb209674e34a77235d2547c968fecc04ae10"}
Nov 04 01:33:18.870578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.870499    5797 scope.go:122] "RemoveContainer" containerID="ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91"
Nov 04 01:33:18.881598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.881577    5797 scope.go:122] "RemoveContainer" containerID="ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91"
Nov 04 01:33:18.881901 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:18.881879    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91\": container with ID starting with ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91 not found: ID does not exist" containerID="ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91"
Nov 04 01:33:18.881946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.881914    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91"} err="failed to get container status \"ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91\": rpc error: code = NotFound desc = could not find container \"ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91\": container with ID starting with ab6ceea37046496e893dd3c5daa0c00b93aed57c46069301834610b833f27c91 not found: ID does not exist"
Nov 04 01:33:18.891265 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.891236    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"]
Nov 04 01:33:18.894479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:18.894459    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-lvdcj"]
Nov 04 01:33:19.098085 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:19.098013    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:19.971817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:19.971777    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:20.458189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:20.458156    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:20.478446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:20.478413    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" path="/var/lib/kubelet/pods/6b6ded4e-6d49-4cda-bbb9-243a999d05e9/volumes"
Nov 04 01:33:20.671517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:20.671484    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:20.881473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:20.881422    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:21.497738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:21.497706    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:22.064760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:22.064725    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:22.292045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:22.291982    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:22.563215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:22.563180    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:22.789860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:22.789823    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"]
Nov 04 01:33:22.915344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:22.915283    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy" containerID="cri-o://e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf" gracePeriod=30
Nov 04 01:33:22.952526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:22.950891    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:23.170794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:23.170716    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:23.744413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:23.744369    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:24.031308 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.031231    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"]
Nov 04 01:33:24.047228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.047199    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:24.244539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.244508    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:24.390749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.390712    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"]
Nov 04 01:33:24.391221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.391204    5797 cpu_manager.go:414] "containerMap: removing container" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy"
Nov 04 01:33:24.391292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.391223    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy"
Nov 04 01:33:24.391292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.391255    5797 cpu_manager.go:414] "containerMap: removing container" podUID="d61b0760-a032-45c1-b3d9-b95049996d1f" containerName="revoke-keys"
Nov 04 01:33:24.391292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.391261    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61b0760-a032-45c1-b3d9-b95049996d1f" containerName="revoke-keys"
Nov 04 01:33:24.391409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.391341    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6ded4e-6d49-4cda-bbb9-243a999d05e9" containerName="istio-proxy"
Nov 04 01:33:24.391409 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.391355    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61b0760-a032-45c1-b3d9-b95049996d1f" containerName="revoke-keys"
Nov 04 01:33:24.396467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.396446    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.406024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.405966    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"]
Nov 04 01:33:24.406483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.406460    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:33:24.406673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.406656    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:33:24.527782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.527752    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.527958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.527801    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppf5v\" (UniqueName: \"kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.527958 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.527884    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.629010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.628961    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.629208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.629048    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppf5v\" (UniqueName: \"kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.629208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.629088    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.629623 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.629601    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.631720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.631694    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.650942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.650874    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppf5v\" (UniqueName: \"kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v\") pod \"maas-api-revoke-sub-46767210-v2n5p\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.713789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.713756    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:24.998420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:24.998387    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"]
Nov 04 01:33:25.001124 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:33:25.001095    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336335a1_1cef_4b09_8ba5_900979b089de.slice/crio-996a3cd4bdd0fdec77e5a6b501d9a32cd8c6e671c9cf75b46458391eafbb483b WatchSource:0}: Error finding container 996a3cd4bdd0fdec77e5a6b501d9a32cd8c6e671c9cf75b46458391eafbb483b: Status 404 returned error can't find the container with id 996a3cd4bdd0fdec77e5a6b501d9a32cd8c6e671c9cf75b46458391eafbb483b
Nov 04 01:33:25.001352 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:25.001264    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:25.314649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:25.314612    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:25.974009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:25.962904    5797 generic.go:356] "Generic (PLEG): container finished" podID="336335a1-1cef-4b09-8ba5-900979b089de" containerID="136ddeae08679ed243784970f0b23026c5ceff1a68932c870571054836265641" exitCode=0
Nov 04 01:33:25.974009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:25.963184    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p" event={"ID":"336335a1-1cef-4b09-8ba5-900979b089de","Type":"ContainerDied","Data":"136ddeae08679ed243784970f0b23026c5ceff1a68932c870571054836265641"}
Nov 04 01:33:25.974009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:25.963236    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p" event={"ID":"336335a1-1cef-4b09-8ba5-900979b089de","Type":"ContainerStarted","Data":"996a3cd4bdd0fdec77e5a6b501d9a32cd8c6e671c9cf75b46458391eafbb483b"}
Nov 04 01:33:25.999390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:25.999355    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:26.260066 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:26.257532    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:33:26.785349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:26.785310    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:27.125862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.125832    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:27.200899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.200862    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca\") pod \"336335a1-1cef-4b09-8ba5-900979b089de\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") "
Nov 04 01:33:27.201139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.201067    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v\" (UniqueName: \"kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v\") pod \"336335a1-1cef-4b09-8ba5-900979b089de\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") "
Nov 04 01:33:27.201139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.201115    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca\") pod \"336335a1-1cef-4b09-8ba5-900979b089de\" (UID: \"336335a1-1cef-4b09-8ba5-900979b089de\") "
Nov 04 01:33:27.201282 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.201228    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca" pod "336335a1-1cef-4b09-8ba5-900979b089de" (UID: "336335a1-1cef-4b09-8ba5-900979b089de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:27.201579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.201558    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:27.203207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.203177    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca" pod "336335a1-1cef-4b09-8ba5-900979b089de" (UID: "336335a1-1cef-4b09-8ba5-900979b089de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:27.203341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.203287    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v" pod "336335a1-1cef-4b09-8ba5-900979b089de" (UID: "336335a1-1cef-4b09-8ba5-900979b089de"). InnerVolumeSpecName "kube-api-access-ppf5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:27.302863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.302831    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppf5v\" (UniqueName: \"kubernetes.io/projected/336335a1-1cef-4b09-8ba5-900979b089de-kube-api-access-ppf5v\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:27.302863 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.302858    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336335a1-1cef-4b09-8ba5-900979b089de-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:27.482655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.482624    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:27.762727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.762654    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"]
Nov 04 01:33:27.762874 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.762852    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery" containerID="cri-o://5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493" gracePeriod=30
Nov 04 01:33:27.981966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.981589    5797 generic.go:356] "Generic (PLEG): container finished" podID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerID="5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493" exitCode=0
Nov 04 01:33:27.981966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.981880    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" event={"ID":"f6f7bd07-f7a6-4817-9644-63ceeff7a207","Type":"ContainerDied","Data":"5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493"}
Nov 04 01:33:27.983799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.983761    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p" event={"ID":"336335a1-1cef-4b09-8ba5-900979b089de","Type":"ContainerDied","Data":"996a3cd4bdd0fdec77e5a6b501d9a32cd8c6e671c9cf75b46458391eafbb483b"}
Nov 04 01:33:27.983935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.983814    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"
Nov 04 01:33:27.983935 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:27.983819    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="996a3cd4bdd0fdec77e5a6b501d9a32cd8c6e671c9cf75b46458391eafbb483b"
Nov 04 01:33:28.045108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.045086    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:33:28.212764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.212738    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:33:28.213304 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213233    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213299    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213366    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213434    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213468    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213519    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213613 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213559    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n\") pod \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\" (UID: \"f6f7bd07-f7a6-4817-9644-63ceeff7a207\") "
Nov 04 01:33:28.213781 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.213743    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:28.214152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.214128    5797 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-ca-configmap\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.216467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.216425    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:28.216598 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.216581    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:28.216659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.216601    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:28.217447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.217419    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "kube-api-access-2kb9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:28.217612 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.217591    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:28.217774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.217754    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert" pod "f6f7bd07-f7a6-4817-9644-63ceeff7a207" (UID: "f6f7bd07-f7a6-4817-9644-63ceeff7a207"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:28.315402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315319    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.315402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315376    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.315402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315400    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.315650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315462    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.315650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315591    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.315785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315756    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.315858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315797    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:28.315858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315808    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:28.315858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315810    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.316047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315894    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:28.316047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315913    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.316047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.315946    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo\") pod \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\" (UID: \"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38\") "
Nov 04 01:33:28.316047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316040    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:28.316279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316085    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:28.316574 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316546    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316578    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316596    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316611    5797 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-cacerts\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316625    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316638    5797 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-kubeconfig\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316652    5797 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f7bd07-f7a6-4817-9644-63ceeff7a207-local-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.316678 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316668    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.317110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316683    5797 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6f7bd07-f7a6-4817-9644-63ceeff7a207-istio-csr-dns-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.317110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316697    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kb9n\" (UniqueName: \"kubernetes.io/projected/f6f7bd07-f7a6-4817-9644-63ceeff7a207-kube-api-access-2kb9n\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.317110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.316713    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.317827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.317798    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:28.318109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.318079    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "kube-api-access-4kf72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:28.318210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.318130    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:33:28.318210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.318142    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy" pod "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" (UID: "f4f6a452-5b44-4a9c-b3d3-f7508f95fa38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:28.328047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.328021    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:28.417542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.417508    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.417542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.417535    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kf72\" (UniqueName: \"kubernetes.io/projected/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-kube-api-access-4kf72\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.417542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.417547    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.417758 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.417556    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:28.978686 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.978642    5797 scope.go:122] "RemoveContainer" containerID="e55aa749411c66a5a0591a9b9c169b85d38be01a8823dc26957ce7b5f4802711"
Nov 04 01:33:28.991230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.991201    5797 scope.go:122] "RemoveContainer" containerID="d595ed7a8b5ac2b57f689e9b87546d08eb7beb1dcbcc134f0f1ce2f772d9f091"
Nov 04 01:33:28.993396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.993368    5797 generic.go:356] "Generic (PLEG): container finished" podID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerID="e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf" exitCode=0
Nov 04 01:33:28.993501 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.993464    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"
Nov 04 01:33:28.993567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.993510    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" event={"ID":"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38","Type":"ContainerDied","Data":"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"}
Nov 04 01:33:28.993609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.993572    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9" event={"ID":"f4f6a452-5b44-4a9c-b3d3-f7508f95fa38","Type":"ContainerDied","Data":"bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac"}
Nov 04 01:33:28.993609 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.993604    5797 scope.go:122] "RemoveContainer" containerID="e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"
Nov 04 01:33:28.995550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.995524    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2" event={"ID":"f6f7bd07-f7a6-4817-9644-63ceeff7a207","Type":"ContainerDied","Data":"ff661e098e0124788d4aa56d301fb628f0469ba1a4dae21129c3d167bbefe47d"}
Nov 04 01:33:28.995662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:28.995637    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"
Nov 04 01:33:29.007836 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.007813    5797 scope.go:122] "RemoveContainer" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:33:29.008280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.008262    5797 scope.go:122] "RemoveContainer" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:33:29.018713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.018694    5797 scope.go:122] "RemoveContainer" containerID="e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"
Nov 04 01:33:29.019072 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.019045    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf\": container with ID starting with e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf not found: ID does not exist" containerID="e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"
Nov 04 01:33:29.019170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.019084    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"} err="failed to get container status \"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf\": rpc error: code = NotFound desc = could not find container \"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf\": container with ID starting with e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf not found: ID does not exist"
Nov 04 01:33:29.019170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.019107    5797 scope.go:122] "RemoveContainer" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:33:29.019281 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.019208    5797 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9_openshift-ingress_f4f6a452-5b44-4a9c-b3d3-f7508f95fa38_0 in pod sandbox bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac from index: no such id: '38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3'" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:33:29.019281 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.019237    5797 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9_openshift-ingress_f4f6a452-5b44-4a9c-b3d3-f7508f95fa38_0 in pod sandbox bdb0c418e26b64abcd7bbe3b4091d4d91d5e532596eae2abea7798abe56e70ac from index: no such id: '38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3'" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:33:29.019281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.019254    5797 scope.go:122] "RemoveContainer" containerID="e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"
Nov 04 01:33:29.019571 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.019363    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3\": container with ID starting with 38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3 not found: ID does not exist" containerID="38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"
Nov 04 01:33:29.019571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.019397    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3"} err="failed to get container status \"38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3\": rpc error: code = NotFound desc = could not find container \"38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3\": container with ID starting with 38b9ca9808848a35d02a416687a429ebe35e8908c84167b573b49009811f3fa3 not found: ID does not exist"
Nov 04 01:33:29.019571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.019417    5797 scope.go:122] "RemoveContainer" containerID="5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493"
Nov 04 01:33:29.022693 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.022669    5797 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf\": rpc error: code = NotFound desc = could not find container \"e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf\": container with ID starting with e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf not found: ID does not exist" containerID="e26168b3887b71b6af1b8a9ef719f71c618a660d2b70f61f246390caeab17bdf"
Nov 04 01:33:29.022790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.022696    5797 scope.go:122] "RemoveContainer" containerID="02732a2ed8a59cc0f1358ab9a58260748470eb9e0470c165cf090b3b3986064f"
Nov 04 01:33:29.025528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.025504    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"]
Nov 04 01:33:29.032441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.032410    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" containerID="cri-o://f570b845b338aff9dd32a4c3840b2258b56d7f0d74768b23d07ed1caa62c635a" gracePeriod=25
Nov 04 01:33:29.036944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.036919    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjqs2"]
Nov 04 01:33:29.041138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.038828    5797 scope.go:122] "RemoveContainer" containerID="5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493"
Nov 04 01:33:29.041138 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.039153    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493\": container with ID starting with 5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493 not found: ID does not exist" containerID="5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493"
Nov 04 01:33:29.041138 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:29.039174    5797 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493\": rpc error: code = NotFound desc = could not find container \"5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493\": container with ID starting with 5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493 not found: ID does not exist" containerID="5393233d89d4ee7e8374d9e736f0f2395b211174a273cd38b44e7e4dbf421493"
Nov 04 01:33:29.041138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.039193    5797 scope.go:122] "RemoveContainer" containerID="d2ab4a3ea258678cda3de74bc384584f59b17df44cc0a3c687e0fec86b8e365c"
Nov 04 01:33:29.048788 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.048767    5797 scope.go:122] "RemoveContainer" containerID="f92686dab3a0e30de9b6d71aa0c2bfd9053874fc8f5824c04075667cb1541fba"
Nov 04 01:33:29.049961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.049858    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"]
Nov 04 01:33:29.056608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.056585    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0741ef-openshift-default-666b498dc-kw7s9"]
Nov 04 01:33:29.059467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.059448    5797 scope.go:122] "RemoveContainer" containerID="769f2402a588c1a509f6172a6cfc5a57df7746ec285dcce72363d130056bf652"
Nov 04 01:33:29.069401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.069382    5797 scope.go:122] "RemoveContainer" containerID="6ba3f2968d83cbfd5a8d5dd33b7586d851d0e44ef6ad507f6812dea58592bb34"
Nov 04 01:33:29.195438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:29.195413    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:30.088505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:30.088467    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:30.309636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:30.309604    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:30.479098 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:30.479058    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" path="/var/lib/kubelet/pods/f4f6a452-5b44-4a9c-b3d3-f7508f95fa38/volumes"
Nov 04 01:33:30.479643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:30.479620    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" path="/var/lib/kubelet/pods/f6f7bd07-f7a6-4817-9644-63ceeff7a207/volumes"
Nov 04 01:33:30.518178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:30.518144    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:30.911257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:30.911220    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:31.304328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:31.304244    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:31.507877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:31.507836    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:31.702981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:31.702950    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:31.962517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:31.962432    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:32.170307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:32.170273    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:32.370503 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:32.370466    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:32.885106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:32.885063    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main" containerID="cri-o://c58f2e988292e6f78c5f255b7f9f1b6cc705030778045a0bf2244d3ec49d8b56" gracePeriod=45
Nov 04 01:33:32.885280 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:32.885114    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:33.023541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.023502    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"]
Nov 04 01:33:33.030622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.030592    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3562ed-thz9v"]
Nov 04 01:33:33.031893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.031867    5797 generic.go:356] "Generic (PLEG): container finished" podID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerID="c58f2e988292e6f78c5f255b7f9f1b6cc705030778045a0bf2244d3ec49d8b56" exitCode=0
Nov 04 01:33:33.032034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.031942    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" event={"ID":"ba7ac48d-d096-4727-bcf6-02f7e33277af","Type":"ContainerDied","Data":"c58f2e988292e6f78c5f255b7f9f1b6cc705030778045a0bf2244d3ec49d8b56"}
Nov 04 01:33:33.139921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.139861    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:33:33.204003 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.203954    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm\") pod \"ba7ac48d-d096-4727-bcf6-02f7e33277af\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") "
Nov 04 01:33:33.204197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204079    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs\") pod \"ba7ac48d-d096-4727-bcf6-02f7e33277af\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") "
Nov 04 01:33:33.204197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204106    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache\") pod \"ba7ac48d-d096-4727-bcf6-02f7e33277af\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") "
Nov 04 01:33:33.204197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204134    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln\" (UniqueName: \"kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln\") pod \"ba7ac48d-d096-4727-bcf6-02f7e33277af\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") "
Nov 04 01:33:33.204197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204167    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home\") pod \"ba7ac48d-d096-4727-bcf6-02f7e33277af\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") "
Nov 04 01:33:33.204441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204211    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir\") pod \"ba7ac48d-d096-4727-bcf6-02f7e33277af\" (UID: \"ba7ac48d-d096-4727-bcf6-02f7e33277af\") "
Nov 04 01:33:33.204703 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204677    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache" pod "ba7ac48d-d096-4727-bcf6-02f7e33277af" (UID: "ba7ac48d-d096-4727-bcf6-02f7e33277af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:33.204905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204884    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home" pod "ba7ac48d-d096-4727-bcf6-02f7e33277af" (UID: "ba7ac48d-d096-4727-bcf6-02f7e33277af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:33.205020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.204967    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir" pod "ba7ac48d-d096-4727-bcf6-02f7e33277af" (UID: "ba7ac48d-d096-4727-bcf6-02f7e33277af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:33.207133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.207102    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln" pod "ba7ac48d-d096-4727-bcf6-02f7e33277af" (UID: "ba7ac48d-d096-4727-bcf6-02f7e33277af"). InnerVolumeSpecName "kube-api-access-6mqln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:33.207316 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.207229    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm" pod "ba7ac48d-d096-4727-bcf6-02f7e33277af" (UID: "ba7ac48d-d096-4727-bcf6-02f7e33277af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:33.207439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.207414    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs" pod "ba7ac48d-d096-4727-bcf6-02f7e33277af" (UID: "ba7ac48d-d096-4727-bcf6-02f7e33277af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:33.305683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.305649    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:33.305683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.305680    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7ac48d-d096-4727-bcf6-02f7e33277af-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:33.305683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.305690    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:33.305927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.305701    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqln\" (UniqueName: \"kubernetes.io/projected/ba7ac48d-d096-4727-bcf6-02f7e33277af-kube-api-access-6mqln\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:33.305927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.305711    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:33.305927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.305719    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7ac48d-d096-4727-bcf6-02f7e33277af-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:33.582868 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:33.582789    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:34.039413 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.039383    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"
Nov 04 01:33:34.039585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.039428    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf" event={"ID":"ba7ac48d-d096-4727-bcf6-02f7e33277af","Type":"ContainerDied","Data":"41b9dd79179627f6b96945defc7e193414fd9f8b5b313d37352a50eac5c2d301"}
Nov 04 01:33:34.039585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.039477    5797 scope.go:122] "RemoveContainer" containerID="c58f2e988292e6f78c5f255b7f9f1b6cc705030778045a0bf2244d3ec49d8b56"
Nov 04 01:33:34.063731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.063706    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"]
Nov 04 01:33:34.069775 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.069748    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6dac99/simulator-w0-6dac99-kserve-688f9c6d7f-xppnf"]
Nov 04 01:33:34.290914 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.290824    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:34.478008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.477970    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955d5da8-8e4c-4f3f-8101-ffa4792083da" path="/var/lib/kubelet/pods/955d5da8-8e4c-4f3f-8101-ffa4792083da/volumes"
Nov 04 01:33:34.478549 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.478532    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" path="/var/lib/kubelet/pods/ba7ac48d-d096-4727-bcf6-02f7e33277af/volumes"
Nov 04 01:33:34.993174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:34.993141    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:35.684268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:35.684235    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:35.777517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:35.777473    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 01:33:36.388277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:36.388243    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:37.097494 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:37.097464    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:37.787757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:37.787721    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:37.987397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:37.987364    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:38.190066 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:38.190031    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:38.387688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:38.387651    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:38.603495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:38.603406    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:38.605349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:38.605323    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"]
Nov 04 01:33:38.605717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:38.605690    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" containerID="cri-o://2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051" gracePeriod=30
Nov 04 01:33:38.806415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:38.806383    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:39.012443 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:39.012411    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:39.230279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:39.230228    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:39.800548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:39.800427    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:40.331008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:40.330956    5797 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:33:40.331401 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:40.331050    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:33:40.582664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:40.582569    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:41.291889 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:41.291853    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:41.994109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:41.994077    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:42.696901 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.696864    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:42.851804 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.851765    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"]
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852555    5797 cpu_manager.go:414] "containerMap: removing container" podUID="336335a1-1cef-4b09-8ba5-900979b089de" containerName="revoke-keys"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852582    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336335a1-1cef-4b09-8ba5-900979b089de" containerName="revoke-keys"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852617    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852628    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852641    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852649    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852664    5797 cpu_manager.go:414] "containerMap: removing container" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852671    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852690    5797 cpu_manager.go:414] "containerMap: removing container" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852697    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852841    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="336335a1-1cef-4b09-8ba5-900979b089de" containerName="revoke-keys"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852858    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852869    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f7bd07-f7a6-4817-9644-63ceeff7a207" containerName="discovery"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852877    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f6a452-5b44-4a9c-b3d3-f7508f95fa38" containerName="istio-proxy"
Nov 04 01:33:42.852972 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.852888    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7ac48d-d096-4727-bcf6-02f7e33277af" containerName="main"
Nov 04 01:33:42.856599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.856575    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.869478 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.869454    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"]
Nov 04 01:33:42.940108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940076    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940118    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prks\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940279 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940252    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940290    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940317    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940360 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940351    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940408    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940466 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940438    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:42.940533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:42.940522    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041137    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041183    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041210    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041249    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041275    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041306    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041514    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041589    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041607    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041597    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.041801 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041677    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prks\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.042268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.041936    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.042268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.042015    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.042268 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.042083    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.043631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.043605    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.044072 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.044054    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.054153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.054127    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.054368 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.054343    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prks\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks\") pod \"e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") " pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.182676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.182645    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:43.482345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.482320    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"]
Nov 04 01:33:43.484890 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:33:43.484861    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741f980e_bcff_4903_a743_ebc2725cb398.slice/crio-367aef3baf41d0c3c68b510efb1cc0f2c7d4cc0feaa295101a5fb2ddb1ea10d9 WatchSource:0}: Error finding container 367aef3baf41d0c3c68b510efb1cc0f2c7d4cc0feaa295101a5fb2ddb1ea10d9: Status 404 returned error can't find the container with id 367aef3baf41d0c3c68b510efb1cc0f2c7d4cc0feaa295101a5fb2ddb1ea10d9
Nov 04 01:33:43.487616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.487583    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:33:43.487704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.487650    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:33:43.487704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.487677    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:33:43.496902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.496880    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:43.866256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.866223    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.953821    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.953921    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954102    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954151    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954228    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954311    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954344    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954398    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954426    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket\") pod \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\" (UID: \"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe\") "
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954749    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.954885    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.955078    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:43.957683 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.957602    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:43.958619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.958068    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:43.967035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.959059    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:43.984200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.977278    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "kube-api-access-57slq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:43.984200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.977515    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:43.984200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.980211    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:43.985046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:43.985009    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo" pod "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" (UID: "7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065445    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065482    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065495    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065509    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57slq\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-kube-api-access-57slq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065523    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065535    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065548    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.065587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.065563    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:44.077850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.077792    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"]
Nov 04 01:33:44.103011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.089645    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"]
Nov 04 01:33:44.126684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.125566    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"]
Nov 04 01:33:44.188752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.188390    5797 generic.go:356] "Generic (PLEG): container finished" podID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerID="2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051" exitCode=0
Nov 04 01:33:44.188752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.188467    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"
Nov 04 01:33:44.188752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.188545    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" event={"ID":"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe","Type":"ContainerDied","Data":"2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051"}
Nov 04 01:33:44.188752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.188591    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7" event={"ID":"7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe","Type":"ContainerDied","Data":"c732938faae343414fc997ed0ca30cca6769cfd75182b4da2c57558072b29ad1"}
Nov 04 01:33:44.188752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.188614    5797 scope.go:122] "RemoveContainer" containerID="2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051"
Nov 04 01:33:44.193421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.193392    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" event={"ID":"741f980e-bcff-4903-a743-ebc2725cb398","Type":"ContainerStarted","Data":"7bfe47e7d1c95b8f121bcf9c24daffd22e52204cbec20a13a807e829503389aa"}
Nov 04 01:33:44.193521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.193454    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" event={"ID":"741f980e-bcff-4903-a743-ebc2725cb398","Type":"ContainerStarted","Data":"367aef3baf41d0c3c68b510efb1cc0f2c7d4cc0feaa295101a5fb2ddb1ea10d9"}
Nov 04 01:33:44.207978 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.207951    5797 scope.go:122] "RemoveContainer" containerID="f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"
Nov 04 01:33:44.220893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.220827    5797 scope.go:122] "RemoveContainer" containerID="2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051"
Nov 04 01:33:44.221163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.221127    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podStartSLOduration=2.221112957 podStartE2EDuration="2.221112957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:44.217656009 +0000 UTC m=+1811.859562276" watchObservedRunningTime="2026-11-04 01:33:44.221112957 +0000 UTC m=+1811.863019220"
Nov 04 01:33:44.221293 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:44.221265    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051\": container with ID starting with 2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051 not found: ID does not exist" containerID="2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051"
Nov 04 01:33:44.221352 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.221308    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051"} err="failed to get container status \"2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051\": rpc error: code = NotFound desc = could not find container \"2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051\": container with ID starting with 2a1bdbfb7c569e82dc98f13a5d72bfed128fdcda0bfb5dbc2477ccc0c2574051 not found: ID does not exist"
Nov 04 01:33:44.221352 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.221334    5797 scope.go:122] "RemoveContainer" containerID="f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"
Nov 04 01:33:44.222060 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:44.221687    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e\": container with ID starting with f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e not found: ID does not exist" containerID="f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"
Nov 04 01:33:44.222060 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.221719    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e"} err="failed to get container status \"f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e\": rpc error: code = NotFound desc = could not find container \"f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e\": container with ID starting with f1640e0c385d618b76bc4470179f8b56f9b8d1637c38fa19a3e0bf5638e87c9e not found: ID does not exist"
Nov 04 01:33:44.236210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.236169    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"]
Nov 04 01:33:44.238270 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.238250    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ce9e8-openshift-default-866bf89ccc-jh5r7"]
Nov 04 01:33:44.304052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.300903    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:44.493015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.489540    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" path="/var/lib/kubelet/pods/7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe/volumes"
Nov 04 01:33:44.531093 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:44.531017    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:45.105397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:45.105351    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:45.189014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:45.188909    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:45.189222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:45.189058    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 01:33:45.189222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:45.189103    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 01:33:45.319968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:45.319924    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:45.778643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:45.778603    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 01:33:46.006827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:46.006789    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:46.183677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:46.183645    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 01:33:46.184138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:46.183700    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 01:33:46.582377 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:46.582289    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:33:46.703869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:46.703844    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:33:46.810748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:46.810715    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:47.183340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:47.183301    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 01:33:47.183563 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:47.183366    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 01:33:47.620859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:47.620829    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:48.183176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:48.183145    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 01:33:48.183387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:48.183195    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 01:33:48.472075 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:48.471412    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:49.184191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.184163    5797 patch_prober.go:29] interesting pod/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 01:33:49.184588 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.184214    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 01:33:49.299447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.299357    5797 generic.go:356] "Generic (PLEG): container finished" podID="148e1b22-7bb1-4399-950c-66a0608a202a" containerID="f570b845b338aff9dd32a4c3840b2258b56d7f0d74768b23d07ed1caa62c635a" exitCode=0
Nov 04 01:33:49.299616 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.299442    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" event={"ID":"148e1b22-7bb1-4399-950c-66a0608a202a","Type":"ContainerDied","Data":"f570b845b338aff9dd32a4c3840b2258b56d7f0d74768b23d07ed1caa62c635a"}
Nov 04 01:33:49.330820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.330780    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:49.385069 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.385044    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:33:49.495339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.495310    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz\" (UniqueName: \"kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz\") pod \"148e1b22-7bb1-4399-950c-66a0608a202a\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") "
Nov 04 01:33:49.495523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.495483    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume\") pod \"148e1b22-7bb1-4399-950c-66a0608a202a\" (UID: \"148e1b22-7bb1-4399-950c-66a0608a202a\") "
Nov 04 01:33:49.495826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.495796    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume" pod "148e1b22-7bb1-4399-950c-66a0608a202a" (UID: "148e1b22-7bb1-4399-950c-66a0608a202a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:49.496352 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.496326    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e1b22-7bb1-4399-950c-66a0608a202a-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:49.497624 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.497596    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz" pod "148e1b22-7bb1-4399-950c-66a0608a202a" (UID: "148e1b22-7bb1-4399-950c-66a0608a202a"). InnerVolumeSpecName "kube-api-access-dpmrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:49.596985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.596896    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmrz\" (UniqueName: \"kubernetes.io/projected/148e1b22-7bb1-4399-950c-66a0608a202a-kube-api-access-dpmrz\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:49.993753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:49.993717    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:50.307722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.307643    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"
Nov 04 01:33:50.307722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.307670    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld" event={"ID":"148e1b22-7bb1-4399-950c-66a0608a202a","Type":"ContainerDied","Data":"cb566d115e7db076eec593cfcbb32a825f7237ff39f986f0bd86bbb99204add2"}
Nov 04 01:33:50.308375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.307740    5797 scope.go:122] "RemoveContainer" containerID="f570b845b338aff9dd32a4c3840b2258b56d7f0d74768b23d07ed1caa62c635a"
Nov 04 01:33:50.333148 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.333123    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"]
Nov 04 01:33:50.341212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.341184    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6dac99-5654849fff-cq7ld"]
Nov 04 01:33:50.405733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.405697    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:50.481604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.481569    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" path="/var/lib/kubelet/pods/148e1b22-7bb1-4399-950c-66a0608a202a/volumes"
Nov 04 01:33:50.604315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.604226    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:50.897860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.897824    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:50.908156 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.908127    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:50.908723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.908699    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:50.909458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:50.909442    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:33:51.396727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:51.396695    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"]
Nov 04 01:33:51.410903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:51.410873    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"]
Nov 04 01:33:51.913116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:51.913050    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:52.113871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:52.113797    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:54.912442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:54.912406    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:55.108899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:55.108869    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:55.663014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:55.662947    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:55.983366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:55.983266    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:56.156843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.156801    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"]
Nov 04 01:33:56.157105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.157073    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy" containerID="cri-o://52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88" gracePeriod=30
Nov 04 01:33:56.171620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.171588    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"]
Nov 04 01:33:56.171807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.171784    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy" containerID="cri-o://9b39d5d52f7079be593160b6018baeef5932dac8b2dda6754eb75f59ae4f8fba" gracePeriod=30
Nov 04 01:33:56.175122 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.175095    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"]
Nov 04 01:33:56.175326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.175299    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy" containerID="cri-o://7bfe47e7d1c95b8f121bcf9c24daffd22e52204cbec20a13a807e829503389aa" gracePeriod=30
Nov 04 01:33:56.289695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.289498    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:56.294467 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:56.294432    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:33:56.398068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.398026    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing" containerID="cri-o://66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238" gracePeriod=25
Nov 04 01:33:56.411750 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.411709    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing" containerID="cri-o://d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f" gracePeriod=25
Nov 04 01:33:56.692135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.692103    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:33:56.719691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.719666    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:33:56.779420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.779386    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume\") pod \"c78d06ce-2464-4370-928b-3935d9667f66\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") "
Nov 04 01:33:56.779618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.779459    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8\" (UniqueName: \"kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8\") pod \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") "
Nov 04 01:33:56.779618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.779505    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg\" (UniqueName: \"kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg\") pod \"c78d06ce-2464-4370-928b-3935d9667f66\" (UID: \"c78d06ce-2464-4370-928b-3935d9667f66\") "
Nov 04 01:33:56.779618 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.779525    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume\") pod \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\" (UID: \"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8\") "
Nov 04 01:33:56.779876 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.779841    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume" pod "c78d06ce-2464-4370-928b-3935d9667f66" (UID: "c78d06ce-2464-4370-928b-3935d9667f66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:56.780032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.780000    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume" pod "fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" (UID: "fb840530-c49b-4bad-b7b7-d94b4c7dd0c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:33:56.781764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.781738    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8" pod "fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" (UID: "fb840530-c49b-4bad-b7b7-d94b4c7dd0c8"). InnerVolumeSpecName "kube-api-access-ddps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:56.781854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.781775    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg" pod "c78d06ce-2464-4370-928b-3935d9667f66" (UID: "c78d06ce-2464-4370-928b-3935d9667f66"). InnerVolumeSpecName "kube-api-access-wf2tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:56.880975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.880938    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddps8\" (UniqueName: \"kubernetes.io/projected/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-kube-api-access-ddps8\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:56.880975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.880970    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf2tg\" (UniqueName: \"kubernetes.io/projected/c78d06ce-2464-4370-928b-3935d9667f66-kube-api-access-wf2tg\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:56.880975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.880980    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:56.881257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:56.881027    5797 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78d06ce-2464-4370-928b-3935d9667f66-plugins-config-volume\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:57.369142 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.369104    5797 generic.go:356] "Generic (PLEG): container finished" podID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerID="66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238" exitCode=0
Nov 04 01:33:57.369611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.369174    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"
Nov 04 01:33:57.369611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.369189    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" event={"ID":"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8","Type":"ContainerDied","Data":"66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238"}
Nov 04 01:33:57.369611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.369253    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh" event={"ID":"fb840530-c49b-4bad-b7b7-d94b4c7dd0c8","Type":"ContainerDied","Data":"81b185f816e6b318394e40293b7205ea60741ef90b000231321e12c45ad134d6"}
Nov 04 01:33:57.369611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.369286    5797 scope.go:122] "RemoveContainer" containerID="66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238"
Nov 04 01:33:57.371290 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.371072    5797 generic.go:356] "Generic (PLEG): container finished" podID="c78d06ce-2464-4370-928b-3935d9667f66" containerID="d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f" exitCode=0
Nov 04 01:33:57.371290 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.371262    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" event={"ID":"c78d06ce-2464-4370-928b-3935d9667f66","Type":"ContainerDied","Data":"d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f"}
Nov 04 01:33:57.371426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.371306    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd" event={"ID":"c78d06ce-2464-4370-928b-3935d9667f66","Type":"ContainerDied","Data":"73f15b44b66edea1d9930feb29b9f2f885c064c2ddf49865b654deb2e78af034"}
Nov 04 01:33:57.371426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.371375    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"
Nov 04 01:33:57.393129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.393023    5797 scope.go:122] "RemoveContainer" containerID="66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238"
Nov 04 01:33:57.393734 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:57.393703    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238\": container with ID starting with 66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238 not found: ID does not exist" containerID="66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238"
Nov 04 01:33:57.393830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.393744    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238"} err="failed to get container status \"66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238\": rpc error: code = NotFound desc = could not find container \"66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238\": container with ID starting with 66d042e32abec2b1e04a8eb247d3fb9e8910f0c48fc327a174d6174ea132a238 not found: ID does not exist"
Nov 04 01:33:57.393830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.393769    5797 scope.go:122] "RemoveContainer" containerID="d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f"
Nov 04 01:33:57.405110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.405088    5797 scope.go:122] "RemoveContainer" containerID="d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f"
Nov 04 01:33:57.405451 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:33:57.405427    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f\": container with ID starting with d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f not found: ID does not exist" containerID="d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f"
Nov 04 01:33:57.405543 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.405464    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f"} err="failed to get container status \"d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f\": rpc error: code = NotFound desc = could not find container \"d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f\": container with ID starting with d85c0ee8a85a8b4b80f5782805ec98a8c61e50b004f7d3ff1fe52910d33bd93f not found: ID does not exist"
Nov 04 01:33:57.412725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.412690    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"]
Nov 04 01:33:57.419723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.419700    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1036e8-657cdcd7d6-ncmjh"]
Nov 04 01:33:57.433785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.433753    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"]
Nov 04 01:33:57.437530 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:57.437466    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1036e8-95b9f58dc-8jzpd"]
Nov 04 01:33:58.488037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:58.486739    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78d06ce-2464-4370-928b-3935d9667f66" path="/var/lib/kubelet/pods/c78d06ce-2464-4370-928b-3935d9667f66/volumes"
Nov 04 01:33:58.488037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:58.487208    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" path="/var/lib/kubelet/pods/fb840530-c49b-4bad-b7b7-d94b4c7dd0c8/volumes"
Nov 04 01:33:58.614050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:58.614013    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:33:59.136946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.136906    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" containerID="cri-o://4faf549968dcd2a67a29f135e372c2d2e7d1232e456701aceed7c0c42814ac54" gracePeriod=45
Nov 04 01:33:59.141229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.141196    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main" containerID="cri-o://df914983beeed3953e76d4ed81d2971ae6124bb8ea350379582b40d71ee6a3d2" gracePeriod=45
Nov 04 01:33:59.182561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.182492    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main" containerID="cri-o://ef67371c91905bc82fa20063649d1294cb5cf9c730f0b667182fabaac0438fe5" gracePeriod=45
Nov 04 01:33:59.442738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.442199    5797 generic.go:356] "Generic (PLEG): container finished" podID="c70573f5-5de4-4121-a727-c65799a4da86" containerID="4faf549968dcd2a67a29f135e372c2d2e7d1232e456701aceed7c0c42814ac54" exitCode=0
Nov 04 01:33:59.442738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.442385    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" event={"ID":"c70573f5-5de4-4121-a727-c65799a4da86","Type":"ContainerDied","Data":"4faf549968dcd2a67a29f135e372c2d2e7d1232e456701aceed7c0c42814ac54"}
Nov 04 01:33:59.445954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.445016    5797 generic.go:356] "Generic (PLEG): container finished" podID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerID="ef67371c91905bc82fa20063649d1294cb5cf9c730f0b667182fabaac0438fe5" exitCode=0
Nov 04 01:33:59.445954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.445062    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" event={"ID":"9a49d108-2491-4c6c-808d-3b4fd1015e88","Type":"ContainerDied","Data":"ef67371c91905bc82fa20063649d1294cb5cf9c730f0b667182fabaac0438fe5"}
Nov 04 01:33:59.447272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.447247    5797 generic.go:356] "Generic (PLEG): container finished" podID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerID="df914983beeed3953e76d4ed81d2971ae6124bb8ea350379582b40d71ee6a3d2" exitCode=0
Nov 04 01:33:59.447390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.447293    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" event={"ID":"29348c5f-115b-4823-9a13-fe0c34d6765d","Type":"ContainerDied","Data":"df914983beeed3953e76d4ed81d2971ae6124bb8ea350379582b40d71ee6a3d2"}
Nov 04 01:33:59.501023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.500926    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:33:59.503667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.503645    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:33:59.506202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.506176    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:33:59.523704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523671    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache\") pod \"29348c5f-115b-4823-9a13-fe0c34d6765d\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") "
Nov 04 01:33:59.523859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523739    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs\" (UniqueName: \"kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs\") pod \"c70573f5-5de4-4121-a727-c65799a4da86\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") "
Nov 04 01:33:59.523859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523770    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home\") pod \"29348c5f-115b-4823-9a13-fe0c34d6765d\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") "
Nov 04 01:33:59.523859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523800    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache\") pod \"c70573f5-5de4-4121-a727-c65799a4da86\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") "
Nov 04 01:33:59.523859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523836    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj\" (UniqueName: \"kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj\") pod \"29348c5f-115b-4823-9a13-fe0c34d6765d\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") "
Nov 04 01:33:59.524186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523866    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir\") pod \"9a49d108-2491-4c6c-808d-3b4fd1015e88\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") "
Nov 04 01:33:59.524186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523918    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home\") pod \"9a49d108-2491-4c6c-808d-3b4fd1015e88\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") "
Nov 04 01:33:59.524186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524014    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache" pod "29348c5f-115b-4823-9a13-fe0c34d6765d" (UID: "29348c5f-115b-4823-9a13-fe0c34d6765d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.524362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524200    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache" pod "c70573f5-5de4-4121-a727-c65799a4da86" (UID: "c70573f5-5de4-4121-a727-c65799a4da86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.524362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.523982    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs\") pod \"9a49d108-2491-4c6c-808d-3b4fd1015e88\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") "
Nov 04 01:33:59.524362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524320    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache\") pod \"9a49d108-2491-4c6c-808d-3b4fd1015e88\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") "
Nov 04 01:33:59.524513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524368    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs\" (UniqueName: \"kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs\") pod \"29348c5f-115b-4823-9a13-fe0c34d6765d\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") "
Nov 04 01:33:59.524513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524413    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm\") pod \"c70573f5-5de4-4121-a727-c65799a4da86\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") "
Nov 04 01:33:59.524513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524451    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home\") pod \"c70573f5-5de4-4121-a727-c65799a4da86\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") "
Nov 04 01:33:59.524513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524508    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm\") pod \"9a49d108-2491-4c6c-808d-3b4fd1015e88\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") "
Nov 04 01:33:59.524728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524544    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir\") pod \"c70573f5-5de4-4121-a727-c65799a4da86\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") "
Nov 04 01:33:59.524728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524582    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir\") pod \"29348c5f-115b-4823-9a13-fe0c34d6765d\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") "
Nov 04 01:33:59.524728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524611    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k\" (UniqueName: \"kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k\") pod \"9a49d108-2491-4c6c-808d-3b4fd1015e88\" (UID: \"9a49d108-2491-4c6c-808d-3b4fd1015e88\") "
Nov 04 01:33:59.524728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524681    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7\" (UniqueName: \"kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7\") pod \"c70573f5-5de4-4121-a727-c65799a4da86\" (UID: \"c70573f5-5de4-4121-a727-c65799a4da86\") "
Nov 04 01:33:59.524728 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.524720    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm\") pod \"29348c5f-115b-4823-9a13-fe0c34d6765d\" (UID: \"29348c5f-115b-4823-9a13-fe0c34d6765d\") "
Nov 04 01:33:59.525589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.525266    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.525589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.525291    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.525589 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.525429    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home" pod "29348c5f-115b-4823-9a13-fe0c34d6765d" (UID: "29348c5f-115b-4823-9a13-fe0c34d6765d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.526207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.526182    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir" pod "c70573f5-5de4-4121-a727-c65799a4da86" (UID: "c70573f5-5de4-4121-a727-c65799a4da86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.526419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.526387    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache" pod "9a49d108-2491-4c6c-808d-3b4fd1015e88" (UID: "9a49d108-2491-4c6c-808d-3b4fd1015e88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.526702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.526680    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home" pod "c70573f5-5de4-4121-a727-c65799a4da86" (UID: "c70573f5-5de4-4121-a727-c65799a4da86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.529292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.527205    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home" pod "9a49d108-2491-4c6c-808d-3b4fd1015e88" (UID: "9a49d108-2491-4c6c-808d-3b4fd1015e88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.529292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.527343    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir" pod "29348c5f-115b-4823-9a13-fe0c34d6765d" (UID: "29348c5f-115b-4823-9a13-fe0c34d6765d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.529292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.527417    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir" pod "9a49d108-2491-4c6c-808d-3b4fd1015e88" (UID: "9a49d108-2491-4c6c-808d-3b4fd1015e88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.529292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.528941    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs" pod "c70573f5-5de4-4121-a727-c65799a4da86" (UID: "c70573f5-5de4-4121-a727-c65799a4da86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:59.531423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.531290    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm" pod "9a49d108-2491-4c6c-808d-3b4fd1015e88" (UID: "9a49d108-2491-4c6c-808d-3b4fd1015e88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.531997 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.531517    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj" pod "29348c5f-115b-4823-9a13-fe0c34d6765d" (UID: "29348c5f-115b-4823-9a13-fe0c34d6765d"). InnerVolumeSpecName "kube-api-access-4f5mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:59.531997 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.531604    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm" pod "29348c5f-115b-4823-9a13-fe0c34d6765d" (UID: "29348c5f-115b-4823-9a13-fe0c34d6765d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.531997 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.531648    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm" pod "c70573f5-5de4-4121-a727-c65799a4da86" (UID: "c70573f5-5de4-4121-a727-c65799a4da86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:33:59.533553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.533422    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7" pod "c70573f5-5de4-4121-a727-c65799a4da86" (UID: "c70573f5-5de4-4121-a727-c65799a4da86"). InnerVolumeSpecName "kube-api-access-4mvb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:59.533553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.533427    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs" pod "29348c5f-115b-4823-9a13-fe0c34d6765d" (UID: "29348c5f-115b-4823-9a13-fe0c34d6765d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:59.533692 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.533581    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs" pod "9a49d108-2491-4c6c-808d-3b4fd1015e88" (UID: "9a49d108-2491-4c6c-808d-3b4fd1015e88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:33:59.533794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.533776    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k" pod "9a49d108-2491-4c6c-808d-3b4fd1015e88" (UID: "9a49d108-2491-4c6c-808d-3b4fd1015e88"). InnerVolumeSpecName "kube-api-access-g6f9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:33:59.626654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626621    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a49d108-2491-4c6c-808d-3b4fd1015e88-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626651    5797 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-model-cache\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626661    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29348c5f-115b-4823-9a13-fe0c34d6765d-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626669    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626677    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626684    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626693    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70573f5-5de4-4121-a727-c65799a4da86-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626701    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626710    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6f9k\" (UniqueName: \"kubernetes.io/projected/9a49d108-2491-4c6c-808d-3b4fd1015e88-kube-api-access-g6f9k\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626721    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvb7\" (UniqueName: \"kubernetes.io/projected/c70573f5-5de4-4121-a727-c65799a4da86-kube-api-access-4mvb7\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626730    5797 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-dshm\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626739    5797 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70573f5-5de4-4121-a727-c65799a4da86-tls-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626746    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29348c5f-115b-4823-9a13-fe0c34d6765d-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626754    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5mj\" (UniqueName: \"kubernetes.io/projected/29348c5f-115b-4823-9a13-fe0c34d6765d-kube-api-access-4f5mj\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626761    5797 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-tmp-dir\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:33:59.626891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:33:59.626770    5797 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a49d108-2491-4c6c-808d-3b4fd1015e88-home\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:00.453810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.453726    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"
Nov 04 01:34:00.453810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.453743    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" event={"ID":"c70573f5-5de4-4121-a727-c65799a4da86","Type":"ContainerDied","Data":"c97375e7c3734c80606fe434662bef5a694e39b3997f1dc5df530a943ed3c363"}
Nov 04 01:34:00.453810 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.453794    5797 scope.go:122] "RemoveContainer" containerID="4faf549968dcd2a67a29f135e372c2d2e7d1232e456701aceed7c0c42814ac54"
Nov 04 01:34:00.455344 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.455318    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25" event={"ID":"9a49d108-2491-4c6c-808d-3b4fd1015e88","Type":"ContainerDied","Data":"64a74c69e1b51e414a2bc50b83c43e3296bdf9da8c5e569ae62a2ec1cd78e71e"}
Nov 04 01:34:00.455436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.455351    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"
Nov 04 01:34:00.456823 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.456797    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk" event={"ID":"29348c5f-115b-4823-9a13-fe0c34d6765d","Type":"ContainerDied","Data":"beb2a8272545ae4e66b9b9a34cd9a9eea25a017e0cfa2fbafc93f797280c9455"}
Nov 04 01:34:00.456932 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.456859    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"
Nov 04 01:34:00.466495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.466452    5797 scope.go:122] "RemoveContainer" containerID="ef67371c91905bc82fa20063649d1294cb5cf9c730f0b667182fabaac0438fe5"
Nov 04 01:34:00.475596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.475581    5797 scope.go:122] "RemoveContainer" containerID="df914983beeed3953e76d4ed81d2971ae6124bb8ea350379582b40d71ee6a3d2"
Nov 04 01:34:00.483755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.483724    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:34:00.486510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.486490    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"]
Nov 04 01:34:00.493757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.493735    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/distinct-w1-1036e8-kserve-79f7cc8bdf-bnnjk"]
Nov 04 01:34:00.503249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.503226    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"]
Nov 04 01:34:00.507906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.507883    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/premium-w1-1036e8-kserve-c94f848c8-wr4tt"]
Nov 04 01:34:00.519374 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.519352    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"]
Nov 04 01:34:00.523038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:00.523016    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1036e8/simulator-w1-1036e8-kserve-5c6bcdd9f5-9dj25"]
Nov 04 01:34:01.443957 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.443926    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:34:01.476025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.474133    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" event={"ID":"5d17dd83-848f-480e-b510-3580fab17a1a","Type":"ContainerDied","Data":"52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88"}
Nov 04 01:34:01.476025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.475072    5797 scope.go:122] "RemoveContainer" containerID="52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88"
Nov 04 01:34:01.476025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.475128    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:01.476025 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.475596    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"
Nov 04 01:34:01.478043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.477599    5797 generic.go:356] "Generic (PLEG): container finished" podID="5d17dd83-848f-480e-b510-3580fab17a1a" containerID="52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88" exitCode=0
Nov 04 01:34:01.478043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.477786    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq" event={"ID":"5d17dd83-848f-480e-b510-3580fab17a1a","Type":"ContainerDied","Data":"27bdbb216f2b7d6babebf0e5ccea855cff314d0fbd7a385c2d6a2265fdacd681"}
Nov 04 01:34:01.489300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.489272    5797 generic.go:356] "Generic (PLEG): container finished" podID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerID="9b39d5d52f7079be593160b6018baeef5932dac8b2dda6754eb75f59ae4f8fba" exitCode=0
Nov 04 01:34:01.489475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.489376    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" event={"ID":"e9ab521b-2772-44a2-9f3d-a4acc73076dc","Type":"ContainerDied","Data":"9b39d5d52f7079be593160b6018baeef5932dac8b2dda6754eb75f59ae4f8fba"}
Nov 04 01:34:01.491557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.491518    5797 generic.go:356] "Generic (PLEG): container finished" podID="741f980e-bcff-4903-a743-ebc2725cb398" containerID="7bfe47e7d1c95b8f121bcf9c24daffd22e52204cbec20a13a807e829503389aa" exitCode=0
Nov 04 01:34:01.491706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.491574    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" event={"ID":"741f980e-bcff-4903-a743-ebc2725cb398","Type":"ContainerDied","Data":"7bfe47e7d1c95b8f121bcf9c24daffd22e52204cbec20a13a807e829503389aa"}
Nov 04 01:34:01.496929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.496905    5797 scope.go:122] "RemoveContainer" containerID="52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88"
Nov 04 01:34:01.497292 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:01.497263    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88\": container with ID starting with 52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88 not found: ID does not exist" containerID="52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88"
Nov 04 01:34:01.497422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.497301    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88"} err="failed to get container status \"52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88\": rpc error: code = NotFound desc = could not find container \"52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88\": container with ID starting with 52f12b53511f09fec17bcc119e0906de2ff8bedcc05e964690f11013beee0a88 not found: ID does not exist"
Nov 04 01:34:01.497422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.497325    5797 scope.go:122] "RemoveContainer" containerID="4d588ec3d09b5ef50f8f7768a068ce833d537f49b8ca4b16f29caa5aba12b48d"
Nov 04 01:34:01.508968 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.508943    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:34:01.522688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.522658    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:34:01.545517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545485    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.545673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545533    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.545673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545580    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.545673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545612    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.545673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545640    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.545673 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545670    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.545875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.545839    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.546035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546014    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.546137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546050    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.546137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546081    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.546261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546141    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.546261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546197    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546228    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546264    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546294    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.546431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546304    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.546431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546341    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546371    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.546431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546414    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546448    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546474    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546510    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546546    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546575    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546626    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket\") pod \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\" (UID: \"e9ab521b-2772-44a2-9f3d-a4acc73076dc\") "
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546634    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:34:01.546756 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546675    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks\") pod \"741f980e-bcff-4903-a743-ebc2725cb398\" (UID: \"741f980e-bcff-4903-a743-ebc2725cb398\") "
Nov 04 01:34:01.547216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546789    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:34:01.547216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.546947    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.547412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.547388    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.547749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.547670    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.547845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.547793    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.547985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.547947    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.548233 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.547979    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.548594 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.548546    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.549055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.549035    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.549297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.549277    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.550129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.549421    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.550129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.550096    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:34:01.552110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.551349    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.552291 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.552272    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token\") pod \"5d17dd83-848f-480e-b510-3580fab17a1a\" (UID: \"5d17dd83-848f-480e-b510-3580fab17a1a\") "
Nov 04 01:34:01.552477 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.552091    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.553767 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553748    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741f980e-bcff-4903-a743-ebc2725cb398-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.553913 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553890    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553915    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553930    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553943    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553959    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.553975    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554007    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554020    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554034    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554048    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17dd83-848f-480e-b510-3580fab17a1a-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554062    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.554076 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554076    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.556485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554090    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.556485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554107    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.556485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.554556    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:01.556485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.555272    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:34:01.556485 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.556407    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "kube-api-access-6prks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:01.556924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.556891    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.556924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.556914    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "kube-api-access-9jmbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:01.557107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.557052    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:01.557107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.557058    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:34:01.557230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.557148    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4" pod "5d17dd83-848f-480e-b510-3580fab17a1a" (UID: "5d17dd83-848f-480e-b510-3580fab17a1a"). InnerVolumeSpecName "kube-api-access-f5fg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:01.557230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.557198    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy" pod "741f980e-bcff-4903-a743-ebc2725cb398" (UID: "741f980e-bcff-4903-a743-ebc2725cb398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.557230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.557214    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:01.558499 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.558473    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:34:01.558829 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.558808    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token" pod "e9ab521b-2772-44a2-9f3d-a4acc73076dc" (UID: "e9ab521b-2772-44a2-9f3d-a4acc73076dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:01.655390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655350    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655383    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6prks\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-kube-api-access-6prks\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655395    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655405    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5fg4\" (UniqueName: \"kubernetes.io/projected/5d17dd83-848f-480e-b510-3580fab17a1a-kube-api-access-f5fg4\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655413    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655422    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741f980e-bcff-4903-a743-ebc2725cb398-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655434    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741f980e-bcff-4903-a743-ebc2725cb398-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655442    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741f980e-bcff-4903-a743-ebc2725cb398-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655451    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jmbv\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-kube-api-access-9jmbv\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655459    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17dd83-848f-480e-b510-3580fab17a1a-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655467    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ab521b-2772-44a2-9f3d-a4acc73076dc-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.655651 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.655475    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17dd83-848f-480e-b510-3580fab17a1a-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:01.685326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.685233    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:01.799924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.799883    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"]
Nov 04 01:34:01.804057 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:01.804021    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-jf8fq"]
Nov 04 01:34:02.476751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.476678    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" path="/var/lib/kubelet/pods/29348c5f-115b-4823-9a13-fe0c34d6765d/volumes"
Nov 04 01:34:02.477140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.477126    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" path="/var/lib/kubelet/pods/5d17dd83-848f-480e-b510-3580fab17a1a/volumes"
Nov 04 01:34:02.477523 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.477510    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" path="/var/lib/kubelet/pods/9a49d108-2491-4c6c-808d-3b4fd1015e88/volumes"
Nov 04 01:34:02.478199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.478185    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70573f5-5de4-4121-a727-c65799a4da86" path="/var/lib/kubelet/pods/c70573f5-5de4-4121-a727-c65799a4da86/volumes"
Nov 04 01:34:02.498256 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.498231    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"
Nov 04 01:34:02.498366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.498279    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr" event={"ID":"741f980e-bcff-4903-a743-ebc2725cb398","Type":"ContainerDied","Data":"367aef3baf41d0c3c68b510efb1cc0f2c7d4cc0feaa295101a5fb2ddb1ea10d9"}
Nov 04 01:34:02.498366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.498335    5797 scope.go:122] "RemoveContainer" containerID="7bfe47e7d1c95b8f121bcf9c24daffd22e52204cbec20a13a807e829503389aa"
Nov 04 01:34:02.501115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.501079    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp" event={"ID":"e9ab521b-2772-44a2-9f3d-a4acc73076dc","Type":"ContainerDied","Data":"28115bb6d3ca093f859987c37fccb238c27f43d9f7ebd8971faeb7ae01c28078"}
Nov 04 01:34:02.501218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.501094    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"
Nov 04 01:34:02.508427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.508408    5797 scope.go:122] "RemoveContainer" containerID="9b39d5d52f7079be593160b6018baeef5932dac8b2dda6754eb75f59ae4f8fba"
Nov 04 01:34:02.525515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.525491    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"]
Nov 04 01:34:02.530262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.530232    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-hbrgp"]
Nov 04 01:34:02.542704 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.542682    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"]
Nov 04 01:34:02.549569 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:02.549547    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6dac99-openshift-default-cbfb66ffc-xmpbr"]
Nov 04 01:34:04.356846 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:04.356811    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:04.477009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:04.476964    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741f980e-bcff-4903-a743-ebc2725cb398" path="/var/lib/kubelet/pods/741f980e-bcff-4903-a743-ebc2725cb398/volumes"
Nov 04 01:34:04.477439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:04.477425    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" path="/var/lib/kubelet/pods/e9ab521b-2772-44a2-9f3d-a4acc73076dc/volumes"
Nov 04 01:34:06.109286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:06.109254    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:10.386302 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:10.386271    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:10.592116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:10.592078    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:12.738719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.738684    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"]
Nov 04 01:34:12.739681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739650    5797 cpu_manager.go:414] "containerMap: removing container" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main"
Nov 04 01:34:12.739681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739674    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739691    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739697    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739713    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739719    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739727    5797 cpu_manager.go:414] "containerMap: removing container" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739732    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739742    5797 cpu_manager.go:414] "containerMap: removing container" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739747    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739756    5797 cpu_manager.go:414] "containerMap: removing container" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739760    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739770    5797 cpu_manager.go:414] "containerMap: removing container" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739775    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739782    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739787    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739796    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739800    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739806    5797 cpu_manager.go:414] "containerMap: removing container" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739811    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739817    5797 cpu_manager.go:414] "containerMap: removing container" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy"
Nov 04 01:34:12.739837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739821    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739882    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739890    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739896    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="29348c5f-115b-4823-9a13-fe0c34d6765d" containerName="main"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739904    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8a8ff8-1e23-47d0-8c8a-a26e7394ccfe" containerName="istio-proxy"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739912    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d17dd83-848f-480e-b510-3580fab17a1a" containerName="istio-proxy"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739918    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="148e1b22-7bb1-4399-950c-66a0608a202a" containerName="payload-pre-processing"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739923    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb840530-c49b-4bad-b7b7-d94b4c7dd0c8" containerName="payload-processing"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739930    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78d06ce-2464-4370-928b-3935d9667f66" containerName="payload-pre-processing"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739935    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70573f5-5de4-4121-a727-c65799a4da86" containerName="main"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739939    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="741f980e-bcff-4903-a743-ebc2725cb398" containerName="istio-proxy"
Nov 04 01:34:12.740577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.739945    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a49d108-2491-4c6c-808d-3b4fd1015e88" containerName="main"
Nov 04 01:34:12.745193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.745169    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.761191 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.761164    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1036e8-openshift-default-dockercfg-wbg6m\""
Nov 04 01:34:12.762521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.762492    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"]
Nov 04 01:34:12.763043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.762978    5797 cpu_manager.go:414] "containerMap: removing container" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy"
Nov 04 01:34:12.763161 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.763059    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy"
Nov 04 01:34:12.763225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.763168    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ab521b-2772-44a2-9f3d-a4acc73076dc" containerName="istio-proxy"
Nov 04 01:34:12.766438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.766414    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"]
Nov 04 01:34:12.766559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.766543    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.787273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.787243    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"]
Nov 04 01:34:12.790645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.790619    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"]
Nov 04 01:34:12.794783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.794758    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.804294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.804265    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"]
Nov 04 01:34:12.870105 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870069    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870110    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.870248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870141    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.870346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870264    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.870346 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870293    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.870420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870342    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870396    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870442    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9slq\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870467    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870489    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870564 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870511    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870625 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870591    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.870657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870623    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870644    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870673    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870716    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.870789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870751    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870789 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870775    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870802    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870826    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.870920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870884    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.870920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870905    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6rs\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.871017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870919    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.871017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870937    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.871017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.870954    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.871152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.871044    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.871152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.871068    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d22p\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.971792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971764    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.971792 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971796    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d22p\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971821    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971849    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971882    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971915    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971939    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.971962    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972024    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972304    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9slq\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.972446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972334    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972356    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972446 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972402    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972521    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.972669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972569    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972608    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972638    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972673    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972677    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972732    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972764    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972791    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972833    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972871    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.972885 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972880    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972881    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972943    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.972959    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973068    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973119    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973154    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6rs\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973165    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973181    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973274    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.973348 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973317    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.973936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973419    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.973936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973453    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.973936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973567    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.973936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973656    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.973936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.973827    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.974243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.974067    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.974243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.974191    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.974920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.974874    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.975786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.975755    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.976030 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.975978    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.976110 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.976018    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.976226 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.976208    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.976292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.976271    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.984656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.984632    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d22p\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.984761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.984740    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:12.984856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.984837    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9slq\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.984895 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.984859    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:12.985008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.984966    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:12.985826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:12.985811    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6rs\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs\") pod \"e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") " pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:13.058720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.058651    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:13.085743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.085721    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:13.116052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.115081    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:13.346463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.346431    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"]
Nov 04 01:34:13.348510 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:34:13.348483    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab14f9e_7173_4d21_ad4c_c73c16cf0bbf.slice/crio-f350762805bac8d6fc189b6de466a3375f95cf46906694c67aed8780859416ec WatchSource:0}: Error finding container f350762805bac8d6fc189b6de466a3375f95cf46906694c67aed8780859416ec: Status 404 returned error can't find the container with id f350762805bac8d6fc189b6de466a3375f95cf46906694c67aed8780859416ec
Nov 04 01:34:13.350899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.350864    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.351046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.350942    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.351046 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.351003    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.383138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.383106    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"]
Nov 04 01:34:13.384819 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:34:13.384795    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63961994_1976_40e8_86ea_7a50d29049ee.slice/crio-8623598c62c9c098a5f0e51bdd61a4f2f393ce394cdce8a5d38ac6c716c36807 WatchSource:0}: Error finding container 8623598c62c9c098a5f0e51bdd61a4f2f393ce394cdce8a5d38ac6c716c36807: Status 404 returned error can't find the container with id 8623598c62c9c098a5f0e51bdd61a4f2f393ce394cdce8a5d38ac6c716c36807
Nov 04 01:34:13.387355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.387322    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.387445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.387385    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.387445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.387416    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.427162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.427081    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"]
Nov 04 01:34:13.430049 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:34:13.430025    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466e641f_316f_40e4_ba2e_bae3174a5028.slice/crio-86ca217217b0d697c47972476270d02ff4664336c0db415a6f949c948aa682c6 WatchSource:0}: Error finding container 86ca217217b0d697c47972476270d02ff4664336c0db415a6f949c948aa682c6: Status 404 returned error can't find the container with id 86ca217217b0d697c47972476270d02ff4664336c0db415a6f949c948aa682c6
Nov 04 01:34:13.432925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.432894    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.433065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.432958    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.433134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.433102    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:34:13.576011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.575831    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" event={"ID":"63961994-1976-40e8-86ea-7a50d29049ee","Type":"ContainerStarted","Data":"0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f"}
Nov 04 01:34:13.576011 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.575884    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" event={"ID":"63961994-1976-40e8-86ea-7a50d29049ee","Type":"ContainerStarted","Data":"8623598c62c9c098a5f0e51bdd61a4f2f393ce394cdce8a5d38ac6c716c36807"}
Nov 04 01:34:13.581020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.578353    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" event={"ID":"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf","Type":"ContainerStarted","Data":"e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646"}
Nov 04 01:34:13.581020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.578403    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" event={"ID":"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf","Type":"ContainerStarted","Data":"f350762805bac8d6fc189b6de466a3375f95cf46906694c67aed8780859416ec"}
Nov 04 01:34:13.581020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.580319    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" event={"ID":"466e641f-316f-40e4-ba2e-bae3174a5028","Type":"ContainerStarted","Data":"7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37"}
Nov 04 01:34:13.581020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.580366    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" event={"ID":"466e641f-316f-40e4-ba2e-bae3174a5028","Type":"ContainerStarted","Data":"86ca217217b0d697c47972476270d02ff4664336c0db415a6f949c948aa682c6"}
Nov 04 01:34:13.633954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.633890    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podStartSLOduration=1.633858356 podStartE2EDuration="1.633858356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:13.603374929 +0000 UTC m=+1841.245281179" watchObservedRunningTime="2026-11-04 01:34:13.633858356 +0000 UTC m=+1841.275764607"
Nov 04 01:34:13.634535 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.634492    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podStartSLOduration=1.634479211 podStartE2EDuration="1.634479211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:13.631940648 +0000 UTC m=+1841.273846925" watchObservedRunningTime="2026-11-04 01:34:13.634479211 +0000 UTC m=+1841.276385462"
Nov 04 01:34:13.662298 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:13.662144    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podStartSLOduration=1.662121815 podStartE2EDuration="1.662121815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:13.657575246 +0000 UTC m=+1841.299481500" watchObservedRunningTime="2026-11-04 01:34:13.662121815 +0000 UTC m=+1841.304028068"
Nov 04 01:34:14.064074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.063121    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:14.064074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.063266    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 01:34:14.064074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.063310    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 01:34:14.092021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.091279    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:14.092021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.091416    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 04 01:34:14.092021 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.091459    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 04 01:34:14.126016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.121161    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:14.126016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.122969    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 04 01:34:14.126016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:14.123082    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 04 01:34:15.069014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:15.065495    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 01:34:15.069014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:15.065559    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 01:34:15.089013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:15.088651    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 04 01:34:15.089013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:15.088730    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 04 01:34:15.122017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:15.117330    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 04 01:34:15.122017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:15.117390    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 04 01:34:16.063015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:16.062604    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 01:34:16.063015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:16.062689    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 01:34:16.092015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:16.088561    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 04 01:34:16.092015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:16.088638    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 04 01:34:16.117014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:16.116454    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 04 01:34:16.117014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:16.116518    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 04 01:34:17.064483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:17.064439    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 01:34:17.064790 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:17.064760    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 01:34:17.087433 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:17.087391    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 04 01:34:17.087744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:17.087716    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 04 01:34:17.122024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:17.121419    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 04 01:34:17.122024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:17.121482    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 04 01:34:18.062015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:18.061244    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 01:34:18.062015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:18.061310    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 01:34:18.087451 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:18.087411    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 04 01:34:18.087749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:18.087718    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 04 01:34:18.119513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:18.119470    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 04 01:34:18.119884 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:18.119852    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 04 01:34:19.059463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:19.059422    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 01:34:19.060162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:19.060131    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 01:34:19.087132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:19.087095    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 04 01:34:19.087400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:19.087375    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 04 01:34:19.117013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:19.116369    5797 patch_prober.go:29] interesting pod/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 04 01:34:19.117013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:19.116431    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 04 01:34:20.065525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.065489    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:20.066042 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.065909    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:20.066828 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.066803    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:20.091207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.091177    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:20.091507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.091486    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:20.092247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.092232    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:20.119822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.119795    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:20.120136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.120116    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:20.120880 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.120859    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:20.427339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.427300    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:20.630151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:20.630118    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:21.841980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:21.841942    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:22.178423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.178387    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"]
Nov 04 01:34:22.187415 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.187387    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"]
Nov 04 01:34:22.194091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.194060    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"]
Nov 04 01:34:22.334176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.334139    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:22.733281 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.733234    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy" containerID="cri-o://7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37" gracePeriod=30
Nov 04 01:34:22.733486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.733242    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy" containerID="cri-o://e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646" gracePeriod=30
Nov 04 01:34:22.733486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.733341    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy" containerID="cri-o://0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f" gracePeriod=30
Nov 04 01:34:22.819510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:22.819471    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:23.315296 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:23.315253    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:23.511953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:23.511914    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:24.717151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:24.717114    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-v9np2"]
Nov 04 01:34:24.717559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:24.717332    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager" containerID="cri-o://99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a" gracePeriod=30
Nov 04 01:34:24.973903 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:24.973846    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:34:25.099473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.099442    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q\" (UniqueName: \"kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q\") pod \"fd34cf22-51b7-4c47-969d-d8f364a35a38\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") "
Nov 04 01:34:25.099650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.099492    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert\") pod \"fd34cf22-51b7-4c47-969d-d8f364a35a38\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") "
Nov 04 01:34:25.099650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.099595    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs\") pod \"fd34cf22-51b7-4c47-969d-d8f364a35a38\" (UID: \"fd34cf22-51b7-4c47-969d-d8f364a35a38\") "
Nov 04 01:34:25.102249 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.102210    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert" pod "fd34cf22-51b7-4c47-969d-d8f364a35a38" (UID: "fd34cf22-51b7-4c47-969d-d8f364a35a38"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:34:25.102713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.102692    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs" pod "fd34cf22-51b7-4c47-969d-d8f364a35a38" (UID: "fd34cf22-51b7-4c47-969d-d8f364a35a38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:34:25.103643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.103619    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q" pod "fd34cf22-51b7-4c47-969d-d8f364a35a38" (UID: "fd34cf22-51b7-4c47-969d-d8f364a35a38"). InnerVolumeSpecName "kube-api-access-n6m6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:25.200839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.200804    5797 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-metrics-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:25.200839 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.200838    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6m6q\" (UniqueName: \"kubernetes.io/projected/fd34cf22-51b7-4c47-969d-d8f364a35a38-kube-api-access-n6m6q\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:25.201071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.200852    5797 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd34cf22-51b7-4c47-969d-d8f364a35a38-webhook-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:25.757561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.757529    5797 generic.go:356] "Generic (PLEG): container finished" podID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerID="99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a" exitCode=0
Nov 04 01:34:25.758050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.757585    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerDied","Data":"99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a"}
Nov 04 01:34:25.758050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.757605    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2"
Nov 04 01:34:25.758050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.757627    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8c4d6cc-v9np2" event={"ID":"fd34cf22-51b7-4c47-969d-d8f364a35a38","Type":"ContainerDied","Data":"fa683223a5953341f3a43ef660395c56a1b2ea023876ab099a86c9f5d001c7e3"}
Nov 04 01:34:25.758050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.757650    5797 scope.go:122] "RemoveContainer" containerID="99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a"
Nov 04 01:34:25.769023 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.768982    5797 scope.go:122] "RemoveContainer" containerID="4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"
Nov 04 01:34:25.785951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.785911    5797 scope.go:122] "RemoveContainer" containerID="99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a"
Nov 04 01:34:25.786504 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:25.786480    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a\": container with ID starting with 99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a not found: ID does not exist" containerID="99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a"
Nov 04 01:34:25.786621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.786509    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a"} err="failed to get container status \"99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a\": rpc error: code = NotFound desc = could not find container \"99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a\": container with ID starting with 99ba104221a8c79243df9c96d9ac578d8747ecd7955fa79dbab44977a463092a not found: ID does not exist"
Nov 04 01:34:25.786621 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.786528    5797 scope.go:122] "RemoveContainer" containerID="4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"
Nov 04 01:34:25.786788 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:25.786769    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f\": container with ID starting with 4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f not found: ID does not exist" containerID="4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"
Nov 04 01:34:25.786840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.786793    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f"} err="failed to get container status \"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f\": rpc error: code = NotFound desc = could not find container \"4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f\": container with ID starting with 4bb3c52a49a6e27f9339fe89e028e5cd75026dae1ba8848b42d48a361327943f not found: ID does not exist"
Nov 04 01:34:25.788373 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.788350    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-v9np2"]
Nov 04 01:34:25.792254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:25.792232    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bf8c4d6cc-v9np2"]
Nov 04 01:34:26.288352 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:26.288285    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:34:26.480269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:26.480236    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" path="/var/lib/kubelet/pods/fd34cf22-51b7-4c47-969d-d8f364a35a38/volumes"
Nov 04 01:34:27.037707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:27.037657    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"]
Nov 04 01:34:27.041722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:27.041691    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cbaa93-nxkl9"]
Nov 04 01:34:28.444176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.444152    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:28.540960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.540353    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7151fd7a-5b3c-4001-ad42-cdb515c1425b" path="/var/lib/kubelet/pods/7151fd7a-5b3c-4001-ad42-cdb515c1425b/volumes"
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567315    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567379    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567448    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567486    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567549    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567621    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567672    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567716    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.567748    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs\") pod \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\" (UID: \"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf\") "
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.568827    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.569102    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.569224    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.569627    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.576020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.570062    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.613240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.608149    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "kube-api-access-vj6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:28.632031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.630126    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:34:28.633900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.633869    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:28.634313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.634286    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy" pod "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" (UID: "cab14f9e-7173-4d21-ad4c-c73c16cf0bbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689722    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj6rs\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-kube-api-access-vj6rs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689768    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689786    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689803    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689818    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689833    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689846    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689860    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.689923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.689874    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.744439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.744412    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:28.746861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.746838    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797005    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797065    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797147    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797185    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797221    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797280    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797325    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797366    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.798013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.797427    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token\") pod \"63961994-1976-40e8-86ea-7a50d29049ee\" (UID: \"63961994-1976-40e8-86ea-7a50d29049ee\") "
Nov 04 01:34:28.824036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.812092    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.824036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.812172    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.824036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.812220    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.824036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.812379    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:34:28.824036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.822096    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-b44x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:34:28.824036 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.822151    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-b44x4" podUID="fd4a4e92-a390-4fc8-813d-71dff7e4e07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:34:28.833240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.833097    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.843771 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.843705    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.844772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.844749    5797 generic.go:356] "Generic (PLEG): container finished" podID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerID="e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646" exitCode=0
Nov 04 01:34:28.844936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.844922    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"
Nov 04 01:34:28.845067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.845046    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" event={"ID":"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf","Type":"ContainerDied","Data":"e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646"}
Nov 04 01:34:28.845145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.845103    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc" event={"ID":"cab14f9e-7173-4d21-ad4c-c73c16cf0bbf","Type":"ContainerDied","Data":"f350762805bac8d6fc189b6de466a3375f95cf46906694c67aed8780859416ec"}
Nov 04 01:34:28.845145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.845136    5797 scope.go:122] "RemoveContainer" containerID="e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646"
Nov 04 01:34:28.888022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.885414    5797 generic.go:356] "Generic (PLEG): container finished" podID="466e641f-316f-40e4-ba2e-bae3174a5028" containerID="7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37" exitCode=0
Nov 04 01:34:28.888022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.885595    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" event={"ID":"466e641f-316f-40e4-ba2e-bae3174a5028","Type":"ContainerDied","Data":"7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37"}
Nov 04 01:34:28.888022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.885677    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz" event={"ID":"466e641f-316f-40e4-ba2e-bae3174a5028","Type":"ContainerDied","Data":"86ca217217b0d697c47972476270d02ff4664336c0db415a6f949c948aa682c6"}
Nov 04 01:34:28.888022 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.885811    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"
Nov 04 01:34:28.900012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.896097    5797 generic.go:356] "Generic (PLEG): container finished" podID="63961994-1976-40e8-86ea-7a50d29049ee" containerID="0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f" exitCode=0
Nov 04 01:34:28.900012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.896166    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" event={"ID":"63961994-1976-40e8-86ea-7a50d29049ee","Type":"ContainerDied","Data":"0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f"}
Nov 04 01:34:28.900012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.896212    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx" event={"ID":"63961994-1976-40e8-86ea-7a50d29049ee","Type":"ContainerDied","Data":"8623598c62c9c098a5f0e51bdd61a4f2f393ce394cdce8a5d38ac6c716c36807"}
Nov 04 01:34:28.900012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.896292    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.901956    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902079    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902122    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902156    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902191    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902256    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902302    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902508    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902565    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket\") pod \"466e641f-316f-40e4-ba2e-bae3174a5028\" (UID: \"466e641f-316f-40e4-ba2e-bae3174a5028\") "
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.902968    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.903000    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.903016    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63961994-1976-40e8-86ea-7a50d29049ee-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.903020 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.903030    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.903928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.903044    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.903928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.903060    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63961994-1976-40e8-86ea-7a50d29049ee-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:28.911010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.904267    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.911010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.904789    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.911010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.904979    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.911010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.905574    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:34:28.918182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.914767    5797 scope.go:122] "RemoveContainer" containerID="e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646"
Nov 04 01:34:28.918182 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:28.915918    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646\": container with ID starting with e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646 not found: ID does not exist" containerID="e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646"
Nov 04 01:34:28.918182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.915957    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646"} err="failed to get container status \"e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646\": rpc error: code = NotFound desc = could not find container \"e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646\": container with ID starting with e518ce519b8e9ab3d7410282ebe43b235704407d6e47f11ef7b8d23ae5737646 not found: ID does not exist"
Nov 04 01:34:28.918182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.915980    5797 scope.go:122] "RemoveContainer" containerID="7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37"
Nov 04 01:34:28.918182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.916331    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:28.919245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.919208    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.921597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.921547    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:28.925198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.925163    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:34:28.928459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.928422    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:34:28.943400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.943135    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq" pod "466e641f-316f-40e4-ba2e-bae3174a5028" (UID: "466e641f-316f-40e4-ba2e-bae3174a5028"). InnerVolumeSpecName "kube-api-access-n9slq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:28.944002 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.943939    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "kube-api-access-6d22p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:28.946152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.946126    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:34:28.951009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:28.949135    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token" pod "63961994-1976-40e8-86ea-7a50d29049ee" (UID: "63961994-1976-40e8-86ea-7a50d29049ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023729    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023763    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466e641f-316f-40e4-ba2e-bae3174a5028-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023780    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023798    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d22p\" (UniqueName: \"kubernetes.io/projected/63961994-1976-40e8-86ea-7a50d29049ee-kube-api-access-6d22p\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023814    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023828    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023842    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023856    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9slq\" (UniqueName: \"kubernetes.io/projected/466e641f-316f-40e4-ba2e-bae3174a5028-kube-api-access-n9slq\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023870    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023885    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466e641f-316f-40e4-ba2e-bae3174a5028-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023899    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466e641f-316f-40e4-ba2e-bae3174a5028-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.023913    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63961994-1976-40e8-86ea-7a50d29049ee-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.024299    5797 scope.go:122] "RemoveContainer" containerID="7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37"
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:29.024735    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37\": container with ID starting with 7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37 not found: ID does not exist" containerID="7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37"
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.024770    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37"} err="failed to get container status \"7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37\": rpc error: code = NotFound desc = could not find container \"7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37\": container with ID starting with 7625d12db921d3b2760f6989f5a71b12cdd9a6290e31becb9f1ce7113d2c7b37 not found: ID does not exist"
Nov 04 01:34:29.026320 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.024796    5797 scope.go:122] "RemoveContainer" containerID="0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f"
Nov 04 01:34:29.037596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.037560    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"]
Nov 04 01:34:29.039513 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.039492    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-vchwc"]
Nov 04 01:34:29.039583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.039539    5797 scope.go:122] "RemoveContainer" containerID="0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f"
Nov 04 01:34:29.039883 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:29.039863    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f\": container with ID starting with 0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f not found: ID does not exist" containerID="0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f"
Nov 04 01:34:29.039929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.039892    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f"} err="failed to get container status \"0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f\": rpc error: code = NotFound desc = could not find container \"0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f\": container with ID starting with 0907f7c9ea96f827e237df19f3a9caaac0a10f29779b022218964ede7ff3dc8f not found: ID does not exist"
Nov 04 01:34:29.116058 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.114292    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:29.219034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.217161    5797 scope.go:122] "RemoveContainer" containerID="79a7259bb11df76f6032504ac490e6a15d4718cedee2e6f66098647c4e2c1704"
Nov 04 01:34:29.233221 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.233124    5797 scope.go:122] "RemoveContainer" containerID="d51237da76354521c89e1d51bb3d3de947f0f07043ea017403b429338adc3a58"
Nov 04 01:34:29.243817 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.243797    5797 scope.go:122] "RemoveContainer" containerID="93665e8d732092393eebe33061f2aff1eb4d90c2f881a38e4296b3b2cf9c2e1a"
Nov 04 01:34:29.253540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.253515    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"]
Nov 04 01:34:29.256111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.256076    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-tbjpz"]
Nov 04 01:34:29.273037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.268360    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"]
Nov 04 01:34:29.308136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:29.304932    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1036e8-openshift-default-5c566f4dbf-nbnwx"]
Nov 04 01:34:30.476757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:30.476719    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" path="/var/lib/kubelet/pods/466e641f-316f-40e4-ba2e-bae3174a5028/volumes"
Nov 04 01:34:30.477247 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:30.477232    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63961994-1976-40e8-86ea-7a50d29049ee" path="/var/lib/kubelet/pods/63961994-1976-40e8-86ea-7a50d29049ee/volumes"
Nov 04 01:34:30.477708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:30.477695    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" path="/var/lib/kubelet/pods/cab14f9e-7173-4d21-ad4c-c73c16cf0bbf/volumes"
Nov 04 01:34:31.395186 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:31.395149    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:32.383850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:32.383813    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:34:38.023448 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:38.023405    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"]
Nov 04 01:34:38.026106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:38.026073    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3cddec8-fsbts"]
Nov 04 01:34:38.476355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:34:38.476324    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a89f60-50d5-4e2d-8444-147d39c965e0" path="/var/lib/kubelet/pods/49a89f60-50d5-4e2d-8444-147d39c965e0/volumes"
Nov 04 01:34:56.252009 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:34:56.251953    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:35:00.178939 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:00.178898    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:35:02.466669 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:02.466623    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:35:25.596737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:25.596649    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:35:26.251084 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:35:26.251045    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:35:29.447136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:29.447102    5797 scope.go:122] "RemoveContainer" containerID="bbdc90c4d0b5bd2226abb325af884502e9c76aa12ccdc090b9671bd68514b95e"
Nov 04 01:35:29.456196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:29.456176    5797 scope.go:122] "RemoveContainer" containerID="4cd19bf74b175f7aa1a5052f0d8bba51c773f5686aec9b3857dd8c71eec23739"
Nov 04 01:35:31.450508 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:31.450458    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:35:32.430819 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:35:32.430779    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:35:56.314250 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:35:56.314178    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:36:16.672220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:16.672174    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:36:16.879961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:16.879929    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:36:25.525151 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:25.525119    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:36:25.525721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:25.525691    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-dw6nw" podUID="fe939348-5f96-41ac-b733-2746eb79904d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:36:25.544108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:25.544078    5797 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:36:25.544303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:25.544284    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-848f6bf8bc-x2k4g" podUID="c6b2d0c8-f594-4dc9-b8ff-a3dec860c112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:36:26.251367 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:36:26.251330    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:36:29.479754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:29.479724    5797 scope.go:122] "RemoveContainer" containerID="bea3da2a3447d7c316136480cf6f12f978aad3f1177883ec7aa660b4550b0aa4"
Nov 04 01:36:31.503635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:31.503603    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:36:32.486396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:32.486362    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:36:55.025722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:55.025686    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"]
Nov 04 01:36:55.028820 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:55.028791    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-104ebe0d-2mmn7"]
Nov 04 01:36:56.251048 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:36:56.251011    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:36:56.477137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:36:56.477099    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430d8f65-60f7-48d2-a655-c8ad18d337e9" path="/var/lib/kubelet/pods/430d8f65-60f7-48d2-a655-c8ad18d337e9/volumes"
Nov 04 01:37:03.536257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:03.536198    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:37:06.023458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:06.023400    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"]
Nov 04 01:37:06.025951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:06.025930    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6a6d05-fdm4j"]
Nov 04 01:37:06.477824 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:06.477795    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9fe857-4324-41ab-a08e-aa94fca011d6" path="/var/lib/kubelet/pods/eb9fe857-4324-41ab-a08e-aa94fca011d6/volumes"
Nov 04 01:37:14.024960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:14.024920    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"]
Nov 04 01:37:14.027983 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:14.027956    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e42255f-vb5hh"]
Nov 04 01:37:14.477274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:14.477243    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a6b7fc-d80a-4b3e-bab6-3436034c53b0" path="/var/lib/kubelet/pods/77a6b7fc-d80a-4b3e-bab6-3436034c53b0/volumes"
Nov 04 01:37:25.023540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:25.023503    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"]
Nov 04 01:37:25.028334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:25.028308    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff56a408-sl8n2"]
Nov 04 01:37:26.250608 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:37:26.250568    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:37:26.476660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:26.476630    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747867c1-0420-4a36-96a8-8f343a128f4b" path="/var/lib/kubelet/pods/747867c1-0420-4a36-96a8-8f343a128f4b/volumes"
Nov 04 01:37:31.554925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:31.554889    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:37:32.538859 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:32.538822    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:37:47.924705 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:37:47.924666    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:37:56.250685 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:37:56.250648    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:38:17.033608 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:17.033521    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"]
Nov 04 01:38:17.036420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:17.036394    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568ccf3d-rn258"]
Nov 04 01:38:18.476870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:18.476832    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61b0760-a032-45c1-b3d9-b95049996d1f" path="/var/lib/kubelet/pods/d61b0760-a032-45c1-b3d9-b95049996d1f/volumes"
Nov 04 01:38:26.251735 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:38:26.251697    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:38:28.024675 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.024640    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"]
Nov 04 01:38:28.029325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.029300    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46767210-v2n5p"]
Nov 04 01:38:28.135521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.135491    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"]
Nov 04 01:38:28.135722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.135701    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery" containerID="cri-o://6b122cb32add62c3374fb07da786e2bd874194bfac7e94d39cf08f825cf4db48" gracePeriod=30
Nov 04 01:38:28.337236 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.337200    5797 generic.go:356] "Generic (PLEG): container finished" podID="67e7cd6f-8996-4d26-956b-453701b30f64" containerID="6b122cb32add62c3374fb07da786e2bd874194bfac7e94d39cf08f825cf4db48" exitCode=0
Nov 04 01:38:28.337429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.337371    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" event={"ID":"67e7cd6f-8996-4d26-956b-453701b30f64","Type":"ContainerDied","Data":"6b122cb32add62c3374fb07da786e2bd874194bfac7e94d39cf08f825cf4db48"}
Nov 04 01:38:28.409642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.409613    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:38:28.477292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.477258    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336335a1-1cef-4b09-8ba5-900979b089de" path="/var/lib/kubelet/pods/336335a1-1cef-4b09-8ba5-900979b089de/volumes"
Nov 04 01:38:28.522554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522522    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.522719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522561    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.522719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522593    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.522719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522633    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.522719 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522705    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.522942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522739    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.522942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.522772    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap\") pod \"67e7cd6f-8996-4d26-956b-453701b30f64\" (UID: \"67e7cd6f-8996-4d26-956b-453701b30f64\") "
Nov 04 01:38:28.523570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.523521    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:38:28.525219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.525190    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:38:28.525349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.525318    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:38:28.525422 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.525398    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:38:28.525488 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.525442    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:38:28.525540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.525517    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:38:28.525620 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.525604    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6" pod "67e7cd6f-8996-4d26-956b-453701b30f64" (UID: "67e7cd6f-8996-4d26-956b-453701b30f64"). InnerVolumeSpecName "kube-api-access-5fwx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:38:28.624427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624393    5797 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-ca-configmap\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:28.624427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624421    5797 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-kubeconfig\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:28.624427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624430    5797 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-cacerts\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:28.624794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624440    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:28.624794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624449    5797 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67e7cd6f-8996-4d26-956b-453701b30f64-istio-csr-dns-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:28.624794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624458    5797 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e7cd6f-8996-4d26-956b-453701b30f64-local-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:28.624794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:28.624466    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fwx6\" (UniqueName: \"kubernetes.io/projected/67e7cd6f-8996-4d26-956b-453701b30f64-kube-api-access-5fwx6\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:38:29.347875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.347839    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"
Nov 04 01:38:29.347875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.347865    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r" event={"ID":"67e7cd6f-8996-4d26-956b-453701b30f64","Type":"ContainerDied","Data":"33dd09cc1325f0d7cd7d58bf03e50eb36aa292aa8d7daacf12c3653ab134d766"}
Nov 04 01:38:29.348432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.347928    5797 scope.go:122] "RemoveContainer" containerID="6b122cb32add62c3374fb07da786e2bd874194bfac7e94d39cf08f825cf4db48"
Nov 04 01:38:29.370907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.370881    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"]
Nov 04 01:38:29.375137 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.375112    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r7v8r"]
Nov 04 01:38:29.513306 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.513274    5797 scope.go:122] "RemoveContainer" containerID="69ee276e6ac093fed6070e7722745a8859eadc18fd0ee4df2d91a323bf668aa2"
Nov 04 01:38:29.521897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.521877    5797 scope.go:122] "RemoveContainer" containerID="3dd3f5059549ac964b3ecbd47a9dac90fad78925e0575cfedfef5f0a6d0d6a9b"
Nov 04 01:38:29.529391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.529358    5797 scope.go:122] "RemoveContainer" containerID="63f31d6e2178bc8148e6701a05d32500cc08315f6621ef0a40fc4d6c44076148"
Nov 04 01:38:29.537282 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:29.537252    5797 scope.go:122] "RemoveContainer" containerID="089f1c303d6c4bd8d0c41b6f3377e3c650439f5d6e09e875b3f7818682d86a00"
Nov 04 01:38:30.476699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:30.476665    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" path="/var/lib/kubelet/pods/67e7cd6f-8996-4d26-956b-453701b30f64/volumes"
Nov 04 01:38:31.520737 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:31.520701    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:38:32.593727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:32.593689    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:38:46.616208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:46.616179    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:38:46.739228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:38:46.739197    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:38:56.251488 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:38:56.251451    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:39:20.583879 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:20.583842    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:21.008830 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:21.008796    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:21.206195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:21.206161    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:24.663064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:24.663027    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:24.900714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:24.900679    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:25.872286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:25.872254    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:26.086786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:26.086748    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:26.251946 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:39:26.251862    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:39:26.948516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:26.948482    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:27.893749 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:27.893713    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:28.812887 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:28.812854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:29.570911 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:29.570878    5797 scope.go:122] "RemoveContainer" containerID="dd82cf5e636e520dfb70d97375842a62fd37ed22c3fe8301b86ac1a36851f6b8"
Nov 04 01:39:29.579953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:29.579935    5797 scope.go:122] "RemoveContainer" containerID="136ddeae08679ed243784970f0b23026c5ceff1a68932c870571054836265641"
Nov 04 01:39:29.608943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:29.608916    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:30.410037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:30.409985    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:31.207643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:31.207610    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:31.660787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:31.660752    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:32.019960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:32.019874    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:32.638664 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:32.638633    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:32.845056 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:32.845022    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:33.494655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:33.494622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:34.183727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:34.183698    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:34.944877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:34.944840    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:35.696107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:35.696075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:36.398262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:36.398224    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:37.164546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:37.164504    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:37.880691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:37.880657    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:38.708793 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:38.708758    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:39.508038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:39.507986    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:39.738643 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:39.738609    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:40.296486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:40.296451    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:40.506602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:40.506565    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:40.731325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:40.731291    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:41.385490 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:41.385458    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:42.069726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:42.069693    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:42.270070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:42.270037    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:43.041391 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:43.041357    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:43.787012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:43.786948    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:44.686253 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:44.686220    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:45.486225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:45.486192    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:46.280539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:46.280501    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:46.981149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:46.981116    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:47.672772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:47.672718    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:48.381974 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:48.381916    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:49.175184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:49.175149    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:49.875305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:49.875275    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:50.615709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:50.615675    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:51.290392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:51.290358    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:52.079981 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:52.079944    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:52.793960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:52.793924    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:53.690867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:53.690813    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:54.387717 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:54.387669    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:55.187276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:55.187234    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:55.890483 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:55.890417    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:56.251136 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:39:56.251052    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:39:56.582183 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:56.582100    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:57.282622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:57.282591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:58.165861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:58.165831    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:58.900305 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:58.900267    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:39:59.697486 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:39:59.697447    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:00.393619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:00.393583    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:01.168365 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:01.168327    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:01.880077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:01.880042    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:02.634074 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:02.634041    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:03.386642 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:03.386612    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:04.095165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:04.095130    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:04.790854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:04.790818    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:05.584369 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:05.584336    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:06.264693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:06.264653    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:06.970515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:06.970479    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:07.681392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:07.681356    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:08.386174 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:08.386120    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:09.077356 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:09.077268    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:09.779115 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:09.779081    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:10.469575 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:10.469538    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:11.179980 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.179947    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:11.814533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814498    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"]
Nov 04 01:40:11.814923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814890    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:40:11.814966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814901    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:40:11.814966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814953    5797 cpu_manager.go:414] "containerMap: removing container" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy"
Nov 04 01:40:11.814966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814958    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814968    5797 cpu_manager.go:414] "containerMap: removing container" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814974    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.814984    5797 cpu_manager.go:414] "containerMap: removing container" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815004    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815016    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815022    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815028    5797 cpu_manager.go:414] "containerMap: removing container" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery"
Nov 04 01:40:11.815104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815033    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery"
Nov 04 01:40:11.815378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815109    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="466e641f-316f-40e4-ba2e-bae3174a5028" containerName="istio-proxy"
Nov 04 01:40:11.815378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815118    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="63961994-1976-40e8-86ea-7a50d29049ee" containerName="istio-proxy"
Nov 04 01:40:11.815378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815127    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:40:11.815378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815133    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e7cd6f-8996-4d26-956b-453701b30f64" containerName="discovery"
Nov 04 01:40:11.815378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815141    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:40:11.815378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.815150    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab14f9e-7173-4d21-ad4c-c73c16cf0bbf" containerName="istio-proxy"
Nov 04 01:40:11.818858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.818834    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.826984 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.826958    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:40:11.827208 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.827138    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:40:11.828578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.828554    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"]
Nov 04 01:40:11.831364 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.831337    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.831502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.831389    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhr9t\" (UniqueName: \"kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.831502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.831473    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.872222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.872182    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:11.932016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.931954    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.932016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.932015    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhr9t\" (UniqueName: \"kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.932297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.932092    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.932582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.932561    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.934515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.934494    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:11.944551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:11.944527    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhr9t\" (UniqueName: \"kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t\") pod \"maas-api-revoke-sub-ce900312-m7x48\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:12.139197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:12.139163    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:12.414459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:12.414431    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"]
Nov 04 01:40:12.417315 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:40:12.417280    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3561b5c8_43b8_4670_b254_7bd7161502c8.slice/crio-a726797ecf06beae504c6893797419e21bd48d9d4e6025f00b90e36aa3fdde9c WatchSource:0}: Error finding container a726797ecf06beae504c6893797419e21bd48d9d4e6025f00b90e36aa3fdde9c: Status 404 returned error can't find the container with id a726797ecf06beae504c6893797419e21bd48d9d4e6025f00b90e36aa3fdde9c
Nov 04 01:40:12.578460 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:12.578432    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:12.897121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:12.897084    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48" event={"ID":"3561b5c8-43b8-4670-b254-7bd7161502c8","Type":"ContainerStarted","Data":"d9afa215ae6622f017b46596ffa1c96287abdeaf6719a8234d6716fcf3d4a0c9"}
Nov 04 01:40:12.897506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:12.897146    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48" event={"ID":"3561b5c8-43b8-4670-b254-7bd7161502c8","Type":"ContainerStarted","Data":"a726797ecf06beae504c6893797419e21bd48d9d4e6025f00b90e36aa3fdde9c"}
Nov 04 01:40:12.921120 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:12.921070    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48" podStartSLOduration=1.9210531560000002 podStartE2EDuration="1.921053156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:12.920759997 +0000 UTC m=+2200.562666285" watchObservedRunningTime="2026-11-04 01:40:12.921053156 +0000 UTC m=+2200.562959407"
Nov 04 01:40:13.284853 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:13.284821    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:13.903367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:13.903330    5797 generic.go:356] "Generic (PLEG): container finished" podID="3561b5c8-43b8-4670-b254-7bd7161502c8" containerID="d9afa215ae6622f017b46596ffa1c96287abdeaf6719a8234d6716fcf3d4a0c9" exitCode=0
Nov 04 01:40:13.903783 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:13.903386    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48" event={"ID":"3561b5c8-43b8-4670-b254-7bd7161502c8","Type":"ContainerDied","Data":"d9afa215ae6622f017b46596ffa1c96287abdeaf6719a8234d6716fcf3d4a0c9"}
Nov 04 01:40:13.981244 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:13.981192    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:14.672604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:14.672568    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:15.050875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.050850    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:15.163197 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.163165    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca\") pod \"3561b5c8-43b8-4670-b254-7bd7161502c8\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") "
Nov 04 01:40:15.163375 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.163356    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca\") pod \"3561b5c8-43b8-4670-b254-7bd7161502c8\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") "
Nov 04 01:40:15.163428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.163403    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t\" (UniqueName: \"kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t\") pod \"3561b5c8-43b8-4670-b254-7bd7161502c8\" (UID: \"3561b5c8-43b8-4670-b254-7bd7161502c8\") "
Nov 04 01:40:15.163702 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.163674    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca" pod "3561b5c8-43b8-4670-b254-7bd7161502c8" (UID: "3561b5c8-43b8-4670-b254-7bd7161502c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:40:15.165274 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.165252    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca" pod "3561b5c8-43b8-4670-b254-7bd7161502c8" (UID: "3561b5c8-43b8-4670-b254-7bd7161502c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:40:15.165445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.165421    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t" pod "3561b5c8-43b8-4670-b254-7bd7161502c8" (UID: "3561b5c8-43b8-4670-b254-7bd7161502c8"). InnerVolumeSpecName "kube-api-access-hhr9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:40:15.265176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.265104    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:40:15.265176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.265131    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhr9t\" (UniqueName: \"kubernetes.io/projected/3561b5c8-43b8-4670-b254-7bd7161502c8-kube-api-access-hhr9t\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:40:15.265176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.265141    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3561b5c8-43b8-4670-b254-7bd7161502c8-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:40:15.486397 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.486361    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:15.914077 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.914046    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"
Nov 04 01:40:15.914292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.914076    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48" event={"ID":"3561b5c8-43b8-4670-b254-7bd7161502c8","Type":"ContainerDied","Data":"a726797ecf06beae504c6893797419e21bd48d9d4e6025f00b90e36aa3fdde9c"}
Nov 04 01:40:15.914292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:15.914116    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a726797ecf06beae504c6893797419e21bd48d9d4e6025f00b90e36aa3fdde9c"
Nov 04 01:40:16.181633 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:16.181551    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:16.884010 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:16.883950    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:17.584796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:17.584766    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:18.276277 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:18.276244    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:18.983555 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:18.983513    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:19.688542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:19.688511    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:20.392117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:20.392065    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:21.089269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:21.089235    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:21.774140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:21.774104    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:22.634396 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:22.634348    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:22.836752 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:22.836720    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:23.300796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:23.300762    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:23.555582 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:23.555489    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:24.159146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:24.159111    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:24.987510 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:24.987481    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:25.690323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:25.690279    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:26.252095 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:40:26.252058    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:40:26.393065 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:26.393036    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:27.093874 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:27.093838    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:27.290689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:27.290660    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:27.785381 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:27.785339    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:27.986659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:27.986622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:28.478852 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:28.478824    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:29.278215 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:29.278181    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:29.685008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:29.684954    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:29.884459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:29.884428    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:30.080347 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:30.080267    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:30.282238 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:30.282201    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:30.758843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:30.758810    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:31.705954 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:31.705920    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:32.832897 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:32.832854    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:35.684751 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:35.684718    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:35.907445 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:40:35.907406    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:40:56.251033 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:40:56.250972    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:41:00.948018 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:00.947964    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:01.152426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:01.152389    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:02.147654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:02.147616    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:02.365080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:02.365042    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:03.020652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:03.020620    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:03.684551 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:03.684522    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:04.425431 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:04.425394    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:04.621050 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:04.621015    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:05.440821 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:05.440780    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:06.037765 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:06.037727    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:06.238043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:06.238008    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:06.871276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:06.871243    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:07.542888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:07.542858    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:08.383334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:08.383300    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:09.085294 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:09.085209    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:09.774979 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:09.774941    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:10.590237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:10.590205    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:11.306293 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.306259    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:11.743461 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.743427    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"]
Nov 04 01:41:11.743891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.743870    5797 cpu_manager.go:414] "containerMap: removing container" podUID="3561b5c8-43b8-4670-b254-7bd7161502c8" containerName="revoke-keys"
Nov 04 01:41:11.743891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.743888    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3561b5c8-43b8-4670-b254-7bd7161502c8" containerName="revoke-keys"
Nov 04 01:41:11.744097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.743903    5797 cpu_manager.go:414] "containerMap: removing container" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:41:11.744097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.743908    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:41:11.744097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.744027    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="3561b5c8-43b8-4670-b254-7bd7161502c8" containerName="revoke-keys"
Nov 04 01:41:11.744097 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.744040    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd34cf22-51b7-4c47-969d-d8f364a35a38" containerName="manager"
Nov 04 01:41:11.747228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.747210    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.750782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.750762    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:41:11.753546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.753523    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:41:11.756554 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.756524    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"]
Nov 04 01:41:11.811723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.811685    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.812176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.812118    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpjr\" (UniqueName: \"kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.812176 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.812165    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.913525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.913488    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.913723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.913585    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpjr\" (UniqueName: \"kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.913723 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.913602    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.914178 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.914158    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.915963 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.915937    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:11.925799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:11.925768    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpjr\" (UniqueName: \"kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:12.061844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:12.061751    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:12.077162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:12.077131    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:12.339886 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:12.339863    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"]
Nov 04 01:41:12.342836 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:41:12.342809    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f68c0a_6653_4032_9e8c_5cd7147dbf6a.slice/crio-7a0b921ed821f9791ee88e54ea29089a5cec72f34172fb60741f81321f22230d WatchSource:0}: Error finding container 7a0b921ed821f9791ee88e54ea29089a5cec72f34172fb60741f81321f22230d: Status 404 returned error can't find the container with id 7a0b921ed821f9791ee88e54ea29089a5cec72f34172fb60741f81321f22230d
Nov 04 01:41:12.769748 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:12.769667    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:13.200999 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:13.200959    5797 generic.go:356] "Generic (PLEG): container finished" podID="42f68c0a-6653-4032-9e8c-5cd7147dbf6a" containerID="6c82a3b3235a92e813082ed294b821726b3a8465b25ff86b3589735591d40a7e" exitCode=0
Nov 04 01:41:13.201162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:13.201060    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l" event={"ID":"42f68c0a-6653-4032-9e8c-5cd7147dbf6a","Type":"ContainerDied","Data":"6c82a3b3235a92e813082ed294b821726b3a8465b25ff86b3589735591d40a7e"}
Nov 04 01:41:13.201162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:13.201098    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l" event={"ID":"42f68c0a-6653-4032-9e8c-5cd7147dbf6a","Type":"ContainerStarted","Data":"7a0b921ed821f9791ee88e54ea29089a5cec72f34172fb60741f81321f22230d"}
Nov 04 01:41:13.573773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:13.573676    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:14.288644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.288605    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:14.356946 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.356914    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:14.440550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.440517    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr\" (UniqueName: \"kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr\") pod \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") "
Nov 04 01:41:14.440721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.440601    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca\") pod \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") "
Nov 04 01:41:14.440721 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.440665    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca\") pod \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\" (UID: \"42f68c0a-6653-4032-9e8c-5cd7147dbf6a\") "
Nov 04 01:41:14.441078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.441053    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca" pod "42f68c0a-6653-4032-9e8c-5cd7147dbf6a" (UID: "42f68c0a-6653-4032-9e8c-5cd7147dbf6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:41:14.442679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.442659    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr" pod "42f68c0a-6653-4032-9e8c-5cd7147dbf6a" (UID: "42f68c0a-6653-4032-9e8c-5cd7147dbf6a"). InnerVolumeSpecName "kube-api-access-bbpjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:41:14.442782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.442757    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca" pod "42f68c0a-6653-4032-9e8c-5cd7147dbf6a" (UID: "42f68c0a-6653-4032-9e8c-5cd7147dbf6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:41:14.542313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.542286    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:41:14.542313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.542314    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:41:14.542491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.542329    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpjr\" (UniqueName: \"kubernetes.io/projected/42f68c0a-6653-4032-9e8c-5cd7147dbf6a-kube-api-access-bbpjr\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:41:14.977198 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:14.977153    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:15.211891 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:15.211859    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"
Nov 04 01:41:15.212099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:15.211897    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l" event={"ID":"42f68c0a-6653-4032-9e8c-5cd7147dbf6a","Type":"ContainerDied","Data":"7a0b921ed821f9791ee88e54ea29089a5cec72f34172fb60741f81321f22230d"}
Nov 04 01:41:15.212099 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:15.211946    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a0b921ed821f9791ee88e54ea29089a5cec72f34172fb60741f81321f22230d"
Nov 04 01:41:15.678384 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:15.678349    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:16.389214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:16.389178    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:17.079063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:17.079028    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:17.778630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:17.778603    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:18.483960 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:18.483930    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:19.177241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:19.177200    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:19.970544 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:19.970514    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:20.681767 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:20.681709    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:21.484735 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:21.484677    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:22.042251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:22.042203    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:22.243064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:22.243010    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:22.769641 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:22.769606    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:22.889252 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:22.889171    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"]
Nov 04 01:41:22.890882 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:22.890853    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-042372-2rp4l"]
Nov 04 01:41:22.967710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:22.967680    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:23.389200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:23.389165    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:24.476542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:24.476503    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f68c0a-6653-4032-9e8c-5cd7147dbf6a" path="/var/lib/kubelet/pods/42f68c0a-6653-4032-9e8c-5cd7147dbf6a/volumes"
Nov 04 01:41:26.251367 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:41:26.251328    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:41:31.762128 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:31.762072    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:32.844803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:41:32.844769    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:41:56.250925 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:41:56.250885    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:42:01.733428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:42:01.733388    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:42:26.251110 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:42:26.251067    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:42:31.854437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:42:31.854398    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:42:32.812006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:42:32.811965    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:42:36.699622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:42:36.699579    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:42:36.899426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:42:36.899385    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:42:56.250846 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:42:56.250801    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:43:26.251418 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:43:26.251380    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:43:31.891731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:31.891688    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:33.543559 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:33.543521    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:37.787071 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:37.787027    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:38.404068 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:38.404037    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:38.632577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:38.632540    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:41.509113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:41.509080    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:41.792211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:41.792127    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:42.764732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:42.764691    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:43.017542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:43.017459    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:43.912944 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:43.912904    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:44.919922 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:44.919891    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:45.828797 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:45.828763    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:46.650760 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:46.650651    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:43:46.776698 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:46.776672    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:43:46.937945 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:46.937856    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:47.751230 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:47.751197    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:48.739776 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:48.739744    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:49.740202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:49.740157    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:50.648087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:50.648037    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:51.576218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:51.576180    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:52.419567 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:52.419536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:53.221867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:53.221831    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:53.920048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:53.920010    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:54.738205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:54.738171    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:55.610237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:55.610203    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:56.250443 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:43:56.250402    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:43:56.545971 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:56.545859    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:56.787640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:56.787610    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:57.197849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:57.197818    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:57.483394 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:57.483310    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:57.755822 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:57.755743    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:58.313170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:58.313132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:58.783096 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:58.783059    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:43:59.862827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:43:59.862796    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:00.878596 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:00.878561    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:01.930205 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:01.930162    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:02.722803 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:02.722771    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:03.494850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:03.494819    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:04.201408 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:04.201373    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:04.887385 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:04.887350    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:05.629782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:05.629746    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:06.399188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:06.399153    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:07.167257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:07.167221    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:08.001245 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:08.001206    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:08.799867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:08.799817    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:09.834231 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:09.834194    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:10.136568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:10.136538    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:10.582755 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:10.582670    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:11.288759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:11.288726    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:12.085300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:12.085263    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:12.894190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:12.894155    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:13.592436 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:13.592397    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:14.383402 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:14.383365    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:15.095139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:15.095104    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:15.890251 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:15.890202    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:16.874109 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:16.874075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:17.672218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:17.672185    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:18.382754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:18.382720    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:19.181920 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:19.181883    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:19.894714 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:19.894680    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:20.596411 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:20.596364    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:21.292742 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:21.292701    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:22.099848 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:22.099819    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:22.982813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:22.982782    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:23.701666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:23.701630    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:24.387278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:24.387249    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:25.076301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:25.076264    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:25.779121 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:25.779064    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:26.251280 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:44:26.251245    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:44:26.476101 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:26.476065    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:27.179518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:27.179475    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:27.879054 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:27.878973    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:28.572577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.572543    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:28.690505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.690466    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"]
Nov 04 01:44:28.690951 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.690933    5797 cpu_manager.go:414] "containerMap: removing container" podUID="42f68c0a-6653-4032-9e8c-5cd7147dbf6a" containerName="revoke-keys"
Nov 04 01:44:28.691067 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.690954    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f68c0a-6653-4032-9e8c-5cd7147dbf6a" containerName="revoke-keys"
Nov 04 01:44:28.691136 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.691087    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f68c0a-6653-4032-9e8c-5cd7147dbf6a" containerName="revoke-keys"
Nov 04 01:44:28.694315 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.694292    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.698770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.698743    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:44:28.700519 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.700502    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:44:28.703240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.703215    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"]
Nov 04 01:44:28.750429 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.750401    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.750577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.750438    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkwp\" (UniqueName: \"kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.750629 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.750599    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.851724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.851625    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.851724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.851676    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkwp\" (UniqueName: \"kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.851970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.851828    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.852329 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.852305    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.854297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.854272    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:28.863799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:28.863777    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkwp\" (UniqueName: \"kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp\") pod \"maas-api-revoke-sub-55316fe4-w8c7j\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:29.009366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:29.009327    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:29.281601 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:29.281568    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"]
Nov 04 01:44:29.284796 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:44:29.284765    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c00968_a48a_4862_8121_2a560cd7a4f3.slice/crio-6d6df2ff38e684aa99fe25cb8bc573a12edea981efc3a1afc88070d522957215 WatchSource:0}: Error finding container 6d6df2ff38e684aa99fe25cb8bc573a12edea981efc3a1afc88070d522957215: Status 404 returned error can't find the container with id 6d6df2ff38e684aa99fe25cb8bc573a12edea981efc3a1afc88070d522957215
Nov 04 01:44:29.290189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:29.290166    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:30.079969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:30.079936    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:30.253285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:30.253239    5797 generic.go:356] "Generic (PLEG): container finished" podID="a1c00968-a48a-4862-8121-2a560cd7a4f3" containerID="8c73484ba3a2ffb41ad6bb7d5b7daeec3b26584d46d11bd6af67e406be3da77f" exitCode=0
Nov 04 01:44:30.253487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:30.253341    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j" event={"ID":"a1c00968-a48a-4862-8121-2a560cd7a4f3","Type":"ContainerDied","Data":"8c73484ba3a2ffb41ad6bb7d5b7daeec3b26584d46d11bd6af67e406be3da77f"}
Nov 04 01:44:30.253487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:30.253389    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j" event={"ID":"a1c00968-a48a-4862-8121-2a560cd7a4f3","Type":"ContainerStarted","Data":"6d6df2ff38e684aa99fe25cb8bc573a12edea981efc3a1afc88070d522957215"}
Nov 04 01:44:30.890307 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:30.890268    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:31.393145 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.393119    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:31.478592 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.478553    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp\" (UniqueName: \"kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp\") pod \"a1c00968-a48a-4862-8121-2a560cd7a4f3\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") "
Nov 04 01:44:31.478773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.478653    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca\") pod \"a1c00968-a48a-4862-8121-2a560cd7a4f3\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") "
Nov 04 01:44:31.478773 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.478680    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca\") pod \"a1c00968-a48a-4862-8121-2a560cd7a4f3\" (UID: \"a1c00968-a48a-4862-8121-2a560cd7a4f3\") "
Nov 04 01:44:31.479172 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.479138    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca" pod "a1c00968-a48a-4862-8121-2a560cd7a4f3" (UID: "a1c00968-a48a-4862-8121-2a560cd7a4f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:44:31.480772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.480747    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca" pod "a1c00968-a48a-4862-8121-2a560cd7a4f3" (UID: "a1c00968-a48a-4862-8121-2a560cd7a4f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:44:31.480772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.480763    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp" pod "a1c00968-a48a-4862-8121-2a560cd7a4f3" (UID: "a1c00968-a48a-4862-8121-2a560cd7a4f3"). InnerVolumeSpecName "kube-api-access-2lkwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:44:31.580351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.580266    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lkwp\" (UniqueName: \"kubernetes.io/projected/a1c00968-a48a-4862-8121-2a560cd7a4f3-kube-api-access-2lkwp\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:44:31.580351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.580297    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:44:31.580351 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.580308    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c00968-a48a-4862-8121-2a560cd7a4f3-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:44:31.588665 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.588618    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:31.934166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:31.934133    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:32.264103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:32.264011    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j" event={"ID":"a1c00968-a48a-4862-8121-2a560cd7a4f3","Type":"ContainerDied","Data":"6d6df2ff38e684aa99fe25cb8bc573a12edea981efc3a1afc88070d522957215"}
Nov 04 01:44:32.264103 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:32.264073    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6df2ff38e684aa99fe25cb8bc573a12edea981efc3a1afc88070d522957215"
Nov 04 01:44:32.264284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:32.264171    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"
Nov 04 01:44:32.284163 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:32.284141    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:32.979124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:32.979088    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:33.206216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:33.206185    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:33.772573 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:33.772536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:34.476529 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:34.476494    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:35.186879 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:35.186838    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:35.888108 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:35.888073    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:36.684301 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:36.684269    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:37.379763 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:37.379725    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:38.175690 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:38.175658    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:38.911150 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:38.911117    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:39.532607 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:39.532570    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:39.729424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:39.729392    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:40.017190 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:40.017149    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:40.290603 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:40.290520    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:40.982421 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:40.982385    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:41.680504 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:41.680466    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:42.376747 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:42.376718    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:43.096225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:43.096195    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:43.785615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:43.785568    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:44.483617 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:44.483580    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:44.679825 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:44.679794    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:45.182808 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:45.182774    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:45.608111 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:45.608027    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:45.804388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:45.804354    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:46.171107 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:46.171064    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:46.455237 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:46.455143    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:46.733154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:46.733075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:52.124969 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:52.124929    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:52.327317 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:44:52.327285    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:44:56.251531 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:44:56.251492    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:45:00.308925 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:00.308882    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:00.510187 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:00.510153    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:15.024806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:15.024765    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"]
Nov 04 01:45:15.028039 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:15.028005    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce900312-m7x48"]
Nov 04 01:45:16.477081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:16.477051    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3561b5c8-43b8-4670-b254-7bd7161502c8" path="/var/lib/kubelet/pods/3561b5c8-43b8-4670-b254-7bd7161502c8/volumes"
Nov 04 01:45:16.926985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:16.926952    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:17.188261 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:17.188182    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:18.190542 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:18.190511    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:18.433154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:18.433118    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:18.648044 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:18.647982    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:19.148813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:19.148762    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:19.895254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:19.895216    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:20.828845 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:20.828801    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:21.902442 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:21.902407    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:22.658229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:22.658194    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:22.989541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:22.989453    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:23.195557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:23.195525    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:23.405604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:23.405557    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:23.607228 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:23.607195    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:24.330888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:24.330857    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:25.105546 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:25.105505    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:25.897232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:25.897196    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:26.251784 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:45:26.251691    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:45:26.709871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:26.709834    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:27.134226 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.134192    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"]
Nov 04 01:45:27.134655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.134640    5797 cpu_manager.go:414] "containerMap: removing container" podUID="a1c00968-a48a-4862-8121-2a560cd7a4f3" containerName="revoke-keys"
Nov 04 01:45:27.134707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.134656    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c00968-a48a-4862-8121-2a560cd7a4f3" containerName="revoke-keys"
Nov 04 01:45:27.134745 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.134728    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c00968-a48a-4862-8121-2a560cd7a4f3" containerName="revoke-keys"
Nov 04 01:45:27.138171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.138153    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.145996 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.145970    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:45:27.146123 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.146096    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:45:27.147556 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.147530    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"]
Nov 04 01:45:27.223472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.223436    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.223647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.223542    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.223647 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.223573    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzzn\" (UniqueName: \"kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.324832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.324799    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.325034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.324869    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.325034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.324910    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzzn\" (UniqueName: \"kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.325571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.325550    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.327147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.327123    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.340043 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.340012    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzzn\" (UniqueName: \"kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.449214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.449131    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:27.451782 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.451761    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:27.725495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:27.725415    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"]
Nov 04 01:45:27.732785 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:45:27.732752    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bff954c_53c1_47bf_b723_7ba928c38e07.slice/crio-695a6b3f7a283c23250823a6f61c0f5e3e3b3c1dddeb9b32f1d7b0c584bad747 WatchSource:0}: Error finding container 695a6b3f7a283c23250823a6f61c0f5e3e3b3c1dddeb9b32f1d7b0c584bad747: Status 404 returned error can't find the container with id 695a6b3f7a283c23250823a6f61c0f5e3e3b3c1dddeb9b32f1d7b0c584bad747
Nov 04 01:45:28.103916 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:28.103886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:28.565547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:28.565434    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8" event={"ID":"9bff954c-53c1-47bf-b723-7ba928c38e07","Type":"ContainerStarted","Data":"22901fe6c93914712844b459f3e8d8ac95d5715efc3e316a15e4f94cb3d6252f"}
Nov 04 01:45:28.565547 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:28.565495    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8" event={"ID":"9bff954c-53c1-47bf-b723-7ba928c38e07","Type":"ContainerStarted","Data":"695a6b3f7a283c23250823a6f61c0f5e3e3b3c1dddeb9b32f1d7b0c584bad747"}
Nov 04 01:45:28.584389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:28.584338    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8" podStartSLOduration=1.584321934 podStartE2EDuration="1.584321934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:28.582836947 +0000 UTC m=+2516.224743219" watchObservedRunningTime="2026-11-04 01:45:28.584321934 +0000 UTC m=+2516.226228207"
Nov 04 01:45:28.783387 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:28.783356    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:29.361479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:29.361448    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:29.571553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:29.571525    5797 generic.go:356] "Generic (PLEG): container finished" podID="9bff954c-53c1-47bf-b723-7ba928c38e07" containerID="22901fe6c93914712844b459f3e8d8ac95d5715efc3e316a15e4f94cb3d6252f" exitCode=0
Nov 04 01:45:29.571955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:29.571624    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8" event={"ID":"9bff954c-53c1-47bf-b723-7ba928c38e07","Type":"ContainerDied","Data":"22901fe6c93914712844b459f3e8d8ac95d5715efc3e316a15e4f94cb3d6252f"}
Nov 04 01:45:29.673479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:29.673443    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:30.386590 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.386560    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:30.719982 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.719954    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:30.757426 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.757395    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca\") pod \"9bff954c-53c1-47bf-b723-7ba928c38e07\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") "
Nov 04 01:45:30.757586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.757458    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn\" (UniqueName: \"kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn\") pod \"9bff954c-53c1-47bf-b723-7ba928c38e07\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") "
Nov 04 01:45:30.757586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.757517    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca\") pod \"9bff954c-53c1-47bf-b723-7ba928c38e07\" (UID: \"9bff954c-53c1-47bf-b723-7ba928c38e07\") "
Nov 04 01:45:30.758041 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.758011    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca" pod "9bff954c-53c1-47bf-b723-7ba928c38e07" (UID: "9bff954c-53c1-47bf-b723-7ba928c38e07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:45:30.759699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.759669    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca" pod "9bff954c-53c1-47bf-b723-7ba928c38e07" (UID: "9bff954c-53c1-47bf-b723-7ba928c38e07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:45:30.759699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.759673    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn" pod "9bff954c-53c1-47bf-b723-7ba928c38e07" (UID: "9bff954c-53c1-47bf-b723-7ba928c38e07"). InnerVolumeSpecName "kube-api-access-4bzzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:45:30.859525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.859493    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:45:30.859525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.859521    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzzn\" (UniqueName: \"kubernetes.io/projected/9bff954c-53c1-47bf-b723-7ba928c38e07-kube-api-access-4bzzn\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:45:30.859525 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:30.859534    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bff954c-53c1-47bf-b723-7ba928c38e07-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:45:31.182731 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:31.182695    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:31.582324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:31.582232    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"
Nov 04 01:45:31.582324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:31.582278    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8" event={"ID":"9bff954c-53c1-47bf-b723-7ba928c38e07","Type":"ContainerDied","Data":"695a6b3f7a283c23250823a6f61c0f5e3e3b3c1dddeb9b32f1d7b0c584bad747"}
Nov 04 01:45:31.582515 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:31.582333    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695a6b3f7a283c23250823a6f61c0f5e3e3b3c1dddeb9b32f1d7b0c584bad747"
Nov 04 01:45:31.889170 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:31.889131    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:32.090814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:32.090778    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:32.731860 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:32.731825    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:32.978149 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:32.978115    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:33.472710 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:33.472676    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:34.168780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:34.168747    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:34.876768 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:34.876734    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:35.591662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:35.591617    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:36.278472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:36.278441    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:36.976209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:36.976177    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:37.443031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:37.443000    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:37.646130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:37.646100    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:37.841248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:37.841161    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:38.302152 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:38.302118    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:38.488449 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:38.488416    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"]
Nov 04 01:45:38.490400 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:38.490378    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dbe634-7lcx8"]
Nov 04 01:45:38.514906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:38.514877    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:38.719420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:38.719387    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:38.997419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:38.997318    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:40.476875 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:40.476844    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bff954c-53c1-47bf-b723-7ba928c38e07" path="/var/lib/kubelet/pods/9bff954c-53c1-47bf-b723-7ba928c38e07/volumes"
Nov 04 01:45:43.774577 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:43.774530    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:45.829548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:45.829492    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:46.033722 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:45:46.033686    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:45:56.250982 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:45:56.250944    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:46:00.289000 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:46:00.288958    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:46:26.250764 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:46:26.250725    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:46:27.626928 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:46:27.626878    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:46:28.765212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:46:28.765179    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:46:29.649435 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:46:29.649399    5797 scope.go:122] "RemoveContainer" containerID="d9afa215ae6622f017b46596ffa1c96287abdeaf6719a8234d6716fcf3d4a0c9"
Nov 04 01:46:32.084516 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:46:32.084478    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:46:33.023826 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:46:33.023781    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:46:56.251076 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:46:56.251031    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:47:07.928538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:07.928486    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:08.362135 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:08.362034    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:08.560195 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:08.560155    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:11.350116 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:11.350078    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:11.551229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:11.551192    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:12.705158 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:12.705084    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:12.919241 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:12.919204    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:13.681685 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:13.681650    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:14.583687 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:14.583655    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:15.340649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:15.340616    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:16.201832 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:16.201797    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:16.937735 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:16.937698    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:17.710229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:17.710161    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:18.602340 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:18.602008    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:19.410216 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:19.410184    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:20.114222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:20.114186    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:20.907165 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:20.907131    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:21.828013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:21.827932    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:22.571162 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:22.571131    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:23.288248 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:23.288211    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:23.979297 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:23.979264    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:24.681961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:24.681930    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:25.382655 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:25.382622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:26.093586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:26.093537    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:26.251662 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:47:26.251622    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:47:26.808124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:26.808075    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:27.182323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.182289    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"]
Nov 04 01:47:27.182696 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.182684    5797 cpu_manager.go:414] "containerMap: removing container" podUID="9bff954c-53c1-47bf-b723-7ba928c38e07" containerName="revoke-keys"
Nov 04 01:47:27.182743 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.182698    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bff954c-53c1-47bf-b723-7ba928c38e07" containerName="revoke-keys"
Nov 04 01:47:27.182780 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.182775    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bff954c-53c1-47bf-b723-7ba928c38e07" containerName="revoke-keys"
Nov 04 01:47:27.186104 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.186086    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.192791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.192763    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:47:27.194338 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.194300    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qdgjq\""
Nov 04 01:47:27.195285 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.195263    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"]
Nov 04 01:47:27.321707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.321657    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.321936 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.321791    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.322029 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.321949    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsngp\" (UniqueName: \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.423339 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.423311    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.423539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.423392    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.423539 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.423504    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsngp\" (UniqueName: \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.424081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.424061    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.425709 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.425688    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:27.430181 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:47:27.430161    5797 projected.go:196] Error preparing data for projected volume kube-api-access-wsngp for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 04 01:47:27.430303 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:47:27.430221    5797 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp podName:706c69f1-b553-4724-a242-eeb8b45f866c nodeName:}" failed. No retries permitted until 2026-11-04 01:47:27.930206859 +0000 UTC m=+2635.572113101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wsngp" (UniqueName: "kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp") pod "maas-api-revoke-keys-e2e-ait-39635e51-62xtc" (UID: "706c69f1-b553-4724-a242-eeb8b45f866c") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 04 01:47:27.707333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:27.707293    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:28.030182 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:28.030087    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsngp\" (UniqueName: \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:28.042840 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:28.042804    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsngp\" (UniqueName: \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\") pod \"maas-api-revoke-keys-e2e-ait-39635e51-62xtc\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:28.100502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:28.100471    5797 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc" secret="" err="secret \"maas-api-cleanup-dockercfg-qdgjq\" not found"
Nov 04 01:47:28.100671 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:28.100531    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:28.374212 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:28.374176    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"]
Nov 04 01:47:28.376820 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:47:28.376795    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706c69f1_b553_4724_a242_eeb8b45f866c.slice/crio-3f806390c7acb109518f66c6a1cbfdc43058e9f69522b1921cb05dfd23e4e35a WatchSource:0}: Error finding container 3f806390c7acb109518f66c6a1cbfdc43058e9f69522b1921cb05dfd23e4e35a: Status 404 returned error can't find the container with id 3f806390c7acb109518f66c6a1cbfdc43058e9f69522b1921cb05dfd23e4e35a
Nov 04 01:47:28.661622 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:28.661596    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:29.344676 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:29.344640    5797 generic.go:356] "Generic (PLEG): container finished" podID="706c69f1-b553-4724-a242-eeb8b45f866c" containerID="480929ba92dc064dbb2ee0302916a117cf5c48bfae5c1fdea10acb74b92d46bf" exitCode=0
Nov 04 01:47:29.345073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:29.344725    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc" event={"ID":"706c69f1-b553-4724-a242-eeb8b45f866c","Type":"ContainerDied","Data":"480929ba92dc064dbb2ee0302916a117cf5c48bfae5c1fdea10acb74b92d46bf"}
Nov 04 01:47:29.345073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:29.344787    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc" event={"ID":"706c69f1-b553-4724-a242-eeb8b45f866c","Type":"ContainerStarted","Data":"3f806390c7acb109518f66c6a1cbfdc43058e9f69522b1921cb05dfd23e4e35a"}
Nov 04 01:47:29.488124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:29.488035    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:29.675189 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:29.675156    5797 scope.go:122] "RemoveContainer" containerID="6c82a3b3235a92e813082ed294b821726b3a8465b25ff86b3589735591d40a7e"
Nov 04 01:47:30.375604 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.375574    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:30.490432 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.490407    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:30.557078 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.557046    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\" (UniqueName: \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\") pod \"706c69f1-b553-4724-a242-eeb8b45f866c\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") "
Nov 04 01:47:30.557275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.557113    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca\") pod \"706c69f1-b553-4724-a242-eeb8b45f866c\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") "
Nov 04 01:47:30.557275 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.557156    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca\") pod \"706c69f1-b553-4724-a242-eeb8b45f866c\" (UID: \"706c69f1-b553-4724-a242-eeb8b45f866c\") "
Nov 04 01:47:30.557447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.557421    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca" pod "706c69f1-b553-4724-a242-eeb8b45f866c" (UID: "706c69f1-b553-4724-a242-eeb8b45f866c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:47:30.559378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.559353    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca" pod "706c69f1-b553-4724-a242-eeb8b45f866c" (UID: "706c69f1-b553-4724-a242-eeb8b45f866c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:47:30.559493 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.559415    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp" pod "706c69f1-b553-4724-a242-eeb8b45f866c" (UID: "706c69f1-b553-4724-a242-eeb8b45f866c"). InnerVolumeSpecName "kube-api-access-wsngp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:47:30.658657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.658630    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:47:30.658657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.658656    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706c69f1-b553-4724-a242-eeb8b45f866c-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:47:30.658883 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:30.658666    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsngp\" (UniqueName: \"kubernetes.io/projected/706c69f1-b553-4724-a242-eeb8b45f866c-kube-api-access-wsngp\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:47:31.184200 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:31.184147    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:31.354940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:31.354908    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"
Nov 04 01:47:31.355139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:31.354938    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc" event={"ID":"706c69f1-b553-4724-a242-eeb8b45f866c","Type":"ContainerDied","Data":"3f806390c7acb109518f66c6a1cbfdc43058e9f69522b1921cb05dfd23e4e35a"}
Nov 04 01:47:31.355139 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:31.355009    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f806390c7acb109518f66c6a1cbfdc43058e9f69522b1921cb05dfd23e4e35a"
Nov 04 01:47:31.982282 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:31.982237    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:32.179970 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:32.179932    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:32.674660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:32.674627    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:33.066218 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:33.066115    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:33.392898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:33.392863    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:34.091813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:34.091762    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:34.787112 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:34.787079    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:35.484019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:35.483956    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:36.180660 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:36.180619    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:36.886155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:36.886113    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:37.445854 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:37.445812    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:37.648037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:37.647974    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:37.752232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:37.752153    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"]
Nov 04 01:47:37.757201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:37.757174    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39635e51-62xtc"]
Nov 04 01:47:37.844117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:37.844086    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:38.043027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:38.042932    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:38.477799 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:38.477768    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706c69f1-b553-4724-a242-eeb8b45f866c" path="/var/lib/kubelet/pods/706c69f1-b553-4724-a242-eeb8b45f866c/volumes"
Nov 04 01:47:38.599487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:38.599448    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:39.991164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:39.991117    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:40.340855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.340760    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"]
Nov 04 01:47:40.341495 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.341469    5797 cpu_manager.go:414] "containerMap: removing container" podUID="706c69f1-b553-4724-a242-eeb8b45f866c" containerName="revoke-keys"
Nov 04 01:47:40.341561 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.341501    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706c69f1-b553-4724-a242-eeb8b45f866c" containerName="revoke-keys"
Nov 04 01:47:40.341700 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.341681    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="706c69f1-b553-4724-a242-eeb8b45f866c" containerName="revoke-keys"
Nov 04 01:47:40.345814 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.345792    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.350772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.350738    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-7ba35a-openshift-default-dockercfg-fxkkp\""
Nov 04 01:47:40.356345 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.356320    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"]
Nov 04 01:47:40.464628 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464596    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.464807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464641    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.464807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464661    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.464807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464791    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.465038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464824    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.465038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464844    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwgn\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.465038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464910    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.465038 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.464966    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.465203 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.465056    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.490662 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.490633    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:40.566232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566191    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566247    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566276    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwgn\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566306    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566331    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566369    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566422    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566471    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566795 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566498    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566801    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.566906 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566829    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.567028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566916    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.567028 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.566948    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.567131 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.567083    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.568815 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.568791    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.569034 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.569018    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.578650 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.578624    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.578800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.578782    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwgn\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn\") pod \"e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") " pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.666774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.666746    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:40.689423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.689396    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:40.980343 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.980309    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"]
Nov 04 01:47:40.981879 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:47:40.981851    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0656f85a_b9da_4b28_9b59_08685150fe78.slice/crio-8c776194650520231d7a968c1ebf965a5ff6122c5e9f5f42df5ce57ff2effc91 WatchSource:0}: Error finding container 8c776194650520231d7a968c1ebf965a5ff6122c5e9f5f42df5ce57ff2effc91: Status 404 returned error can't find the container with id 8c776194650520231d7a968c1ebf965a5ff6122c5e9f5f42df5ce57ff2effc91
Nov 04 01:47:40.984140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.984110    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:47:40.984219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.984173    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:47:40.984219 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:40.984202    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:47:41.406677 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:41.406639    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" event={"ID":"0656f85a-b9da-4b28-9b59-08685150fe78","Type":"ContainerStarted","Data":"e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"}
Nov 04 01:47:41.407133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:41.406704    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" event={"ID":"0656f85a-b9da-4b28-9b59-08685150fe78","Type":"ContainerStarted","Data":"8c776194650520231d7a968c1ebf965a5ff6122c5e9f5f42df5ce57ff2effc91"}
Nov 04 01:47:41.434414 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:41.434368    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podStartSLOduration=1.434353623 podStartE2EDuration="1.434353623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:41.431860616 +0000 UTC m=+2649.073766891" watchObservedRunningTime="2026-11-04 01:47:41.434353623 +0000 UTC m=+2649.076259873"
Nov 04 01:47:41.667323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:41.667236    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:42.672720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:42.672682    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:43.417160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:43.417123    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:43.418082 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:43.418060    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:43.760943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:43.760850    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:43.963496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:43.963466    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:44.988667 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:44.988630    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:45.192327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:45.192287    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:45.979202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:45.979164    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:46.783532 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:46.783497    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:47.496141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:47.496107    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:48.285033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:48.284959    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:49.002684 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:49.002646    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:49.686796 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:49.686763    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:50.394091 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:50.394057    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:51.192388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:51.192329    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:51.879720 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:51.879685    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:52.583759 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:52.583725    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:53.283861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:53.283832    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:53.973359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:53.973321    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:54.685045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:54.685006    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:55.494081 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:55.494040    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:56.197827 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:56.197790    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:56.251059 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:47:56.251026    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:47:56.878250 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:56.878217    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:57.517327 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:57.517251    5797 generic.go:356] "Generic (PLEG): container finished" podID="0656f85a-b9da-4b28-9b59-08685150fe78" containerID="e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62" exitCode=0
Nov 04 01:47:57.517517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:57.517322    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" event={"ID":"0656f85a-b9da-4b28-9b59-08685150fe78","Type":"ContainerDied","Data":"e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"}
Nov 04 01:47:57.517726 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:57.517711    5797 scope.go:122] "RemoveContainer" containerID="e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"
Nov 04 01:47:57.519164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:57.519140    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:47:57.519240 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:57.519217    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:47:57.519276 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:57.519255    5797 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 01:47:58.417791 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.417748    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:58.525086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.525051    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" event={"ID":"0656f85a-b9da-4b28-9b59-08685150fe78","Type":"ContainerStarted","Data":"753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba"}
Nov 04 01:47:58.604211 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.604178    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:58.669222 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.667347    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:47:58.672147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.672113    5797 patch_prober.go:29] interesting pod/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 01:47:58.672341 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.672192    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 01:47:58.804035 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:58.803977    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:59.008666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:59.008572    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:47:59.671164 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:59.671123    5797 patch_prober.go:29] interesting pod/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 01:47:59.672017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:59.671770    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 01:47:59.793017 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:47:59.792959    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:00.668014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:00.667625    5797 patch_prober.go:29] interesting pod/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 01:48:00.668014 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:00.667691    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 01:48:00.688654 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:00.688621    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:01.393805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:01.393769    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:01.670073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:01.669413    5797 patch_prober.go:29] interesting pod/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 01:48:01.670073 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:01.669479    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 01:48:02.097679 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:02.097591    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:02.672362 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:02.672327    5797 patch_prober.go:29] interesting pod/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 01:48:02.672548 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:02.672386    5797 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 01:48:02.790636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:02.790601    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:03.480599 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:03.480558    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:03.710114 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:03.710079    5797 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:48:03.711206 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:03.711184    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:48:03.712450 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:03.712427    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:48:04.181269 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:04.181232    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:04.977787 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:04.977750    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:05.675611 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:05.675559    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:06.380835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:06.380799    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:07.074831 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:07.074800    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:07.776975 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:07.776937    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:08.477132 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:08.477097    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:09.174138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:09.174100    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:09.870406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:09.870374    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:10.577357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:10.577326    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:11.275837 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:11.275798    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:12.071155 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:12.071117    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:12.773778 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:12.773736    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:13.467166 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:13.467129    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:14.175689 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:14.175653    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:14.976644 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:14.976608    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:15.772143 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:15.772108    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:16.483372 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:16.483343    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:17.274323 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:17.274285    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:17.976921 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:17.976886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:18.678310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:18.678251    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:19.382627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:19.382596    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:20.075367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:20.075332    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:20.787739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:20.787706    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:21.471424 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:21.471369    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:22.171472 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:22.171417    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:22.985095 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:22.985063    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:23.681856 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:23.681820    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:24.380807 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:24.380771    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:25.079649 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:25.079614    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:25.804697 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:25.804668    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:26.251322 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:48:26.251286    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:48:26.489829 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:26.489800    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:27.176769 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:27.176736    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:27.890080 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:27.890020    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:28.582571 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:28.582537    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:29.392337 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:29.392291    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:30.088303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:30.088264    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:30.790314 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:30.790282    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:31.485447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:31.485414    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:32.182844 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:32.182809    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:32.382858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:32.382818    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:32.878864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:32.878829    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:33.123171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:33.123132    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:33.321378 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:33.321269    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:33.569359 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:33.569328    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:35.619893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:35.619857    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:35.854008 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:35.853969    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:36.471229 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:36.471196    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:37.179707 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:37.179671    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:37.870322 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:37.870295    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:38.579770 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:38.579737    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:39.288688 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:39.288657    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:39.987272 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:39.987238    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:40.678260 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:40.678226    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:41.401458 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:41.401425    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:42.071732 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:42.071699    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:42.781553 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:42.781517    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:43.470953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:43.470922    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:44.178423 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:44.178394    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:44.878015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:44.877974    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:45.577766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:45.577735    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:46.286113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:46.286079    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:46.685855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:46.685830    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:48:46.808818 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:46.808793    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-zcsk7_5083bdb1-d4a2-4162-9b49-e3191fa61637/csi-driver/0.log"
Nov 04 01:48:47.078934 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:47.078845    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:47.766953 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:47.766917    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:48.477310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:48.477268    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:49.170942 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:49.170906    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:49.877355 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:49.877323    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:50.572640 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:50.572599    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:51.291552 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:51.291520    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:51.983873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:51.983836    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:52.686813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:52.686779    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:53.386713 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:53.386680    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:54.076412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:54.076381    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:54.780850 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:54.780812    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:55.476047 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:55.476017    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:56.183015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:56.182955    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:56.251485 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:48:56.251448    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:48:56.888578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:56.888541    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:57.779961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:57.779925    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:58.481615 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:58.481582    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:59.191379 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:59.191341    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:48:59.882210 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:48:59.882177    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:00.586666 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:00.586639    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:01.281657 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:01.281626    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:02.088220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:02.088178    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:02.786585 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:02.786555    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:03.504505 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:03.504471    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:04.192739 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:04.192708    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:04.890064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:04.890030    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:05.591388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:05.591355    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:06.284473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:06.284437    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:06.989106 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:06.989076    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:07.692526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:07.692495    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:08.399766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:08.399732    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:09.105658 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:09.105628    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:09.822480 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:09.822450    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:10.488202 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:10.488163    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:11.192019 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:11.191977    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:11.891122 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:11.891090    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:12.595154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:12.595117    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:13.312593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:13.312559    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:14.013354 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:14.013322    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:14.693576 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:14.693544    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:15.377597 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:15.377561    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:16.071146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:16.071097    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:16.785985 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:16.785939    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:17.484134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:17.484098    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:18.172473 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:18.172430    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:18.879147 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:18.879109    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:19.231809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:19.231736    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:19.587681 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:19.587604    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:20.395009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:20.394950    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:21.193656 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:21.193622    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:21.895331 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:21.895284    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:22.596631 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:22.596598    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:23.290399 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:23.290365    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:23.988380 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:23.988351    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:24.637328 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:24.637297    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:24.833009 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:24.832955    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:25.438201 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:25.438165    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:25.650334 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:25.650301    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:26.250941 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:49:26.250909    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:49:31.724463 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:31.724426    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:31.924812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:31.924776    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:32.024388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.024307    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"]
Nov 04 01:49:32.027273 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.027249    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55316fe4-w8c7j"]
Nov 04 01:49:32.280639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.280559    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:32.476691 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.476657    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:32.480134 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.480106    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c00968-a48a-4862-8121-2a560cd7a4f3" path="/var/lib/kubelet/pods/a1c00968-a48a-4862-8121-2a560cd7a4f3/volumes"
Nov 04 01:49:32.480438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.480419    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"]
Nov 04 01:49:32.480663 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.480642    5797 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" containerID="cri-o://753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba" gracePeriod=30
Nov 04 01:49:32.680324 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:32.680291    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:33.712965 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:33.712931    5797 patch_prober.go:29] interesting pod/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 01:49:33.713389 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:33.713023    5797 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 01:49:36.270015 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:36.267624    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:37.000581 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.000490    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:49:37.792333 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.792306    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:49:37.846950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.846916    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.846968    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847039    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847088    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847171 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847149    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847200    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847366    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:49:37.847457 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847439    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847494    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847555    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data\") pod \"0656f85a-b9da-4b28-9b59-08685150fe78\" (UID: \"0656f85a-b9da-4b28-9b59-08685150fe78\") "
Nov 04 01:49:37.847843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847636    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:49:37.847843 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847759    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:49:37.847973 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847860    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:49:37.848064 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.847980    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:49:37.848160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.848127    5797 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-credential-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.848160 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.848156    5797 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0656f85a-b9da-4b28-9b59-08685150fe78-istiod-ca-cert\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.848288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.848170    5797 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-certs\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.848288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.848183    5797 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-workload-socket\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.848288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.848197    5797 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-data\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.849805 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.849760    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 01:49:37.849943 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.849856    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 01:49:37.850051 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.849968    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "kube-api-access-wcwgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:49:37.850326 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.850302    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token" pod "0656f85a-b9da-4b28-9b59-08685150fe78" (UID: "0656f85a-b9da-4b28-9b59-08685150fe78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:49:37.949500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.949463    5797 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0656f85a-b9da-4b28-9b59-08685150fe78-istio-envoy\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.949500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.949492    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwgn\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-kube-api-access-wcwgn\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.949500 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.949502    5797 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0656f85a-b9da-4b28-9b59-08685150fe78-istio-token\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:37.949738 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:37.949512    5797 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0656f85a-b9da-4b28-9b59-08685150fe78-istio-podinfo\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:38.034392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.034304    5797 generic.go:356] "Generic (PLEG): container finished" podID="0656f85a-b9da-4b28-9b59-08685150fe78" containerID="753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba" exitCode=0
Nov 04 01:49:38.034392 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.034368    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"
Nov 04 01:49:38.034634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.034383    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" event={"ID":"0656f85a-b9da-4b28-9b59-08685150fe78","Type":"ContainerDied","Data":"753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba"}
Nov 04 01:49:38.034634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.034438    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r" event={"ID":"0656f85a-b9da-4b28-9b59-08685150fe78","Type":"ContainerDied","Data":"8c776194650520231d7a968c1ebf965a5ff6122c5e9f5f42df5ce57ff2effc91"}
Nov 04 01:49:38.034634 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.034458    5797 scope.go:122] "RemoveContainer" containerID="753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba"
Nov 04 01:49:38.044325 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.044308    5797 scope.go:122] "RemoveContainer" containerID="e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"
Nov 04 01:49:38.052517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.052498    5797 scope.go:122] "RemoveContainer" containerID="753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba"
Nov 04 01:49:38.052767 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:49:38.052747    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba\": container with ID starting with 753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba not found: ID does not exist" containerID="753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba"
Nov 04 01:49:38.052816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.052775    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba"} err="failed to get container status \"753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba\": rpc error: code = NotFound desc = could not find container \"753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba\": container with ID starting with 753085a9bc8c056d25c39b84e1cbd21517ea3c119ea8d6171598c3ac4e92afba not found: ID does not exist"
Nov 04 01:49:38.052816 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.052791    5797 scope.go:122] "RemoveContainer" containerID="e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"
Nov 04 01:49:38.053140 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:49:38.053115    5797 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62\": container with ID starting with e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62 not found: ID does not exist" containerID="e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"
Nov 04 01:49:38.053239 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.053143    5797 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62"} err="failed to get container status \"e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62\": rpc error: code = NotFound desc = could not find container \"e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62\": container with ID starting with e2c38f61ed3b4f212b44f5ada48ab3878981c0704069518440bb9d848b035e62 not found: ID does not exist"
Nov 04 01:49:38.057753 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.057730    5797 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"]
Nov 04 01:49:38.062645 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.062624    5797 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7ba35a-openshift-default-8f4b6cbfc-h8f8r"]
Nov 04 01:49:38.477813 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:38.477781    5797 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" path="/var/lib/kubelet/pods/0656f85a-b9da-4b28-9b59-08685150fe78/volumes"
Nov 04 01:49:44.780926 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.780889    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"]
Nov 04 01:49:44.781438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.781301    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy"
Nov 04 01:49:44.781438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.781314    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy"
Nov 04 01:49:44.781438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.781366    5797 cpu_manager.go:414] "containerMap: removing container" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy"
Nov 04 01:49:44.781438 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.781372    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy"
Nov 04 01:49:44.781587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.781441    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy"
Nov 04 01:49:44.781587 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.781454    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="0656f85a-b9da-4b28-9b59-08685150fe78" containerName="istio-proxy"
Nov 04 01:49:44.786031 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.786011    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.792610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.792585    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:49:44.792877 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.792673    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8xbr4\""
Nov 04 01:49:44.794881 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.794853    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"]
Nov 04 01:49:44.840033 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.839979    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.840207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.840156    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6l85\" (UniqueName: \"kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.840207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.840198    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.941288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.941248    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6l85\" (UniqueName: \"kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.941288 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.941287    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.941540 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.941333    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.941929 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.941908    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.943630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.943611    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:44.956059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:44.956032    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6l85\" (UniqueName: \"kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85\") pod \"maas-api-revoke-sub-f03629db-cs6kr\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:45.100427 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:45.100348    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:45.375761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:45.375738    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"]
Nov 04 01:49:45.378584 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:49:45.378552    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a341cc3_ab85_4090_826d_c8d56bb849a3.slice/crio-63097c15b5b3b24487fcc5946c6f16f06ae53bf93217cf236673c5797bf2aeaf WatchSource:0}: Error finding container 63097c15b5b3b24487fcc5946c6f16f06ae53bf93217cf236673c5797bf2aeaf: Status 404 returned error can't find the container with id 63097c15b5b3b24487fcc5946c6f16f06ae53bf93217cf236673c5797bf2aeaf
Nov 04 01:49:46.107557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:46.107476    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr" event={"ID":"9a341cc3-ab85-4090-826d-c8d56bb849a3","Type":"ContainerStarted","Data":"34f636f25fca516d663c8e6ca321fdfdeae1cc99f2feb8e5e38d2c85e33b3d00"}
Nov 04 01:49:46.107557 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:46.107527    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr" event={"ID":"9a341cc3-ab85-4090-826d-c8d56bb849a3","Type":"ContainerStarted","Data":"63097c15b5b3b24487fcc5946c6f16f06ae53bf93217cf236673c5797bf2aeaf"}
Nov 04 01:49:46.126570 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:46.126522    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr" podStartSLOduration=2.126509659 podStartE2EDuration="2.126509659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:46.125275226 +0000 UTC m=+2773.767181495" watchObservedRunningTime="2026-11-04 01:49:46.126509659 +0000 UTC m=+2773.768415902"
Nov 04 01:49:47.113864 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:47.113825    5797 generic.go:356] "Generic (PLEG): container finished" podID="9a341cc3-ab85-4090-826d-c8d56bb849a3" containerID="34f636f25fca516d663c8e6ca321fdfdeae1cc99f2feb8e5e38d2c85e33b3d00" exitCode=0
Nov 04 01:49:47.114313 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:47.113892    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr" event={"ID":"9a341cc3-ab85-4090-826d-c8d56bb849a3","Type":"ContainerDied","Data":"34f636f25fca516d663c8e6ca321fdfdeae1cc99f2feb8e5e38d2c85e33b3d00"}
Nov 04 01:49:48.253565 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.253537    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:48.373766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.373671    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca\") pod \"9a341cc3-ab85-4090-826d-c8d56bb849a3\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") "
Nov 04 01:49:48.373766 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.373724    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca\") pod \"9a341cc3-ab85-4090-826d-c8d56bb849a3\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") "
Nov 04 01:49:48.374059 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.373819    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85\" (UniqueName: \"kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85\") pod \"9a341cc3-ab85-4090-826d-c8d56bb849a3\" (UID: \"9a341cc3-ab85-4090-826d-c8d56bb849a3\") "
Nov 04 01:49:48.374133 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.374101    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca" pod "9a341cc3-ab85-4090-826d-c8d56bb849a3" (UID: "9a341cc3-ab85-4090-826d-c8d56bb849a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:49:48.374199 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.374174    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:48.376086 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.376056    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85" pod "9a341cc3-ab85-4090-826d-c8d56bb849a3" (UID: "9a341cc3-ab85-4090-826d-c8d56bb849a3"). InnerVolumeSpecName "kube-api-access-b6l85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:49:48.376214 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.376192    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca" pod "9a341cc3-ab85-4090-826d-c8d56bb849a3" (UID: "9a341cc3-ab85-4090-826d-c8d56bb849a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:49:48.475089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.475055    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a341cc3-ab85-4090-826d-c8d56bb849a3-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:48.475089 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:48.475089    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6l85\" (UniqueName: \"kubernetes.io/projected/9a341cc3-ab85-4090-826d-c8d56bb849a3-kube-api-access-b6l85\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:49:49.125905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:49.125875    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr"
Nov 04 01:49:49.125905 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:49.125893    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cs6kr" event={"ID":"9a341cc3-ab85-4090-826d-c8d56bb849a3","Type":"ContainerDied","Data":"63097c15b5b3b24487fcc5946c6f16f06ae53bf93217cf236673c5797bf2aeaf"}
Nov 04 01:49:49.126144 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:49:49.125961    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63097c15b5b3b24487fcc5946c6f16f06ae53bf93217cf236673c5797bf2aeaf"
Nov 04 01:49:56.251358 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:49:56.251320    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:50:00.797361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.797327    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"]
Nov 04 01:50:00.797812 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.797793    5797 cpu_manager.go:414] "containerMap: removing container" podUID="9a341cc3-ab85-4090-826d-c8d56bb849a3" containerName="revoke-keys"
Nov 04 01:50:00.797861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.797813    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a341cc3-ab85-4090-826d-c8d56bb849a3" containerName="revoke-keys"
Nov 04 01:50:00.797924 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.797910    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a341cc3-ab85-4090-826d-c8d56bb849a3" containerName="revoke-keys"
Nov 04 01:50:00.801196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.801174    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:00.809084 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.809053    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:50:00.809232 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.809058    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8xbr4\""
Nov 04 01:50:00.809977 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.809952    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"]
Nov 04 01:50:00.939800 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.939756    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:00.940024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.939822    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:00.940024 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:00.939896    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264p2\" (UniqueName: \"kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.040733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.040697    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.040733 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.040730    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264p2\" (UniqueName: \"kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.040950 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.040837    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.041425 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.041399    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.043180 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.043159    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.053332 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.053279    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264p2\" (UniqueName: \"kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2\") pod \"maas-api-revoke-sub-0165d7d2-6xbk9\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.116000 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.115968    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:01.389757 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:01.389731    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"]
Nov 04 01:50:01.392074 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:50:01.392046    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6717d5_8761_408d_934f_ff15dee882b8.slice/crio-f4162a5d2275cc6d3e0237a0ae9f574902fd775d0a13152be67d8d0a725f0fc8 WatchSource:0}: Error finding container f4162a5d2275cc6d3e0237a0ae9f574902fd775d0a13152be67d8d0a725f0fc8: Status 404 returned error can't find the container with id f4162a5d2275cc6d3e0237a0ae9f574902fd775d0a13152be67d8d0a725f0fc8
Nov 04 01:50:02.218785 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:02.218744    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9" event={"ID":"2e6717d5-8761-408d-934f-ff15dee882b8","Type":"ContainerStarted","Data":"eb0d169d6b5bb50f7477ab8b60b046759ca514db4fd9652e6faafdd3d1ae5346"}
Nov 04 01:50:02.219292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:02.218829    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9" event={"ID":"2e6717d5-8761-408d-934f-ff15dee882b8","Type":"ContainerStarted","Data":"f4162a5d2275cc6d3e0237a0ae9f574902fd775d0a13152be67d8d0a725f0fc8"}
Nov 04 01:50:02.239652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:02.239603    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9" podStartSLOduration=2.239590217 podStartE2EDuration="2.239590217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:50:02.238416612 +0000 UTC m=+2789.880322879" watchObservedRunningTime="2026-11-04 01:50:02.239590217 +0000 UTC m=+2789.881496466"
Nov 04 01:50:03.226153 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:03.226120    5797 generic.go:356] "Generic (PLEG): container finished" podID="2e6717d5-8761-408d-934f-ff15dee882b8" containerID="eb0d169d6b5bb50f7477ab8b60b046759ca514db4fd9652e6faafdd3d1ae5346" exitCode=0
Nov 04 01:50:03.226578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:03.226170    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9" event={"ID":"2e6717d5-8761-408d-934f-ff15dee882b8","Type":"ContainerDied","Data":"eb0d169d6b5bb50f7477ab8b60b046759ca514db4fd9652e6faafdd3d1ae5346"}
Nov 04 01:50:04.369311 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.369286    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:04.469517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.469486    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca\") pod \"2e6717d5-8761-408d-934f-ff15dee882b8\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") "
Nov 04 01:50:04.469899 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.469877    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca" pod "2e6717d5-8761-408d-934f-ff15dee882b8" (UID: "2e6717d5-8761-408d-934f-ff15dee882b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:50:04.570284 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.570194    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca\") pod \"2e6717d5-8761-408d-934f-ff15dee882b8\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") "
Nov 04 01:50:04.570459 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.570311    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2\" (UniqueName: \"kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2\") pod \"2e6717d5-8761-408d-934f-ff15dee882b8\" (UID: \"2e6717d5-8761-408d-934f-ff15dee882b8\") "
Nov 04 01:50:04.570668 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.570640    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:50:04.572395 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.572372    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca" pod "2e6717d5-8761-408d-934f-ff15dee882b8" (UID: "2e6717d5-8761-408d-934f-ff15dee882b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:50:04.572487 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.572428    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2" pod "2e6717d5-8761-408d-934f-ff15dee882b8" (UID: "2e6717d5-8761-408d-934f-ff15dee882b8"). InnerVolumeSpecName "kube-api-access-264p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:50:04.671521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.671493    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264p2\" (UniqueName: \"kubernetes.io/projected/2e6717d5-8761-408d-934f-ff15dee882b8-kube-api-access-264p2\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:50:04.671521 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:04.671518    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6717d5-8761-408d-934f-ff15dee882b8-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:50:05.176583 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:05.176545    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:50:05.236550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:05.236520    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9"
Nov 04 01:50:05.236550 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:05.236533    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6xbk9" event={"ID":"2e6717d5-8761-408d-934f-ff15dee882b8","Type":"ContainerDied","Data":"f4162a5d2275cc6d3e0237a0ae9f574902fd775d0a13152be67d8d0a725f0fc8"}
Nov 04 01:50:05.236754 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:05.236583    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4162a5d2275cc6d3e0237a0ae9f574902fd775d0a13152be67d8d0a725f0fc8"
Nov 04 01:50:05.376636 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:05.376605    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:50:11.608528 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.608491    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"]
Nov 04 01:50:11.609012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.608972    5797 cpu_manager.go:414] "containerMap: removing container" podUID="2e6717d5-8761-408d-934f-ff15dee882b8" containerName="revoke-keys"
Nov 04 01:50:11.609012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.609005    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6717d5-8761-408d-934f-ff15dee882b8" containerName="revoke-keys"
Nov 04 01:50:11.609129 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.609096    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6717d5-8761-408d-934f-ff15dee882b8" containerName="revoke-keys"
Nov 04 01:50:11.612498 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.612477    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.619126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.619074    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 01:50:11.620366 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.619738    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8xbr4\""
Nov 04 01:50:11.620893 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.620870    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"]
Nov 04 01:50:11.636257 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.636224    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.636419 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.636281    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5z7\" (UniqueName: \"kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.636541 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.636517    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.737390 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.737350    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.737591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.737403    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.737591 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.737452    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5z7\" (UniqueName: \"kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.738016 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.737962    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.739849 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.739824    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.749303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.749282    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5z7\" (UniqueName: \"kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7\") pod \"maas-api-revoke-sub-6b19338f-zqjvr\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:11.928538 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:11.928495    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:12.201898 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:12.201816    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"]
Nov 04 01:50:12.204899 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:50:12.204866    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046d707a_b0ac_4e60_abbd_9fde7350b16a.slice/crio-7596ede49282497ecf24318964c783922434510d95ebd1a458fb0c4f954d1340 WatchSource:0}: Error finding container 7596ede49282497ecf24318964c783922434510d95ebd1a458fb0c4f954d1340: Status 404 returned error can't find the container with id 7596ede49282497ecf24318964c783922434510d95ebd1a458fb0c4f954d1340
Nov 04 01:50:12.270055 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:12.270025    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr" event={"ID":"046d707a-b0ac-4e60-abbd-9fde7350b16a","Type":"ContainerStarted","Data":"7596ede49282497ecf24318964c783922434510d95ebd1a458fb0c4f954d1340"}
Nov 04 01:50:13.276213 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:13.276181    5797 generic.go:356] "Generic (PLEG): container finished" podID="046d707a-b0ac-4e60-abbd-9fde7350b16a" containerID="5aa2091c87df3eadcd7b921d3f5ffcf5608df76c63a5cb5752e321d31d4dafcd" exitCode=0
Nov 04 01:50:13.276602 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:13.276222    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr" event={"ID":"046d707a-b0ac-4e60-abbd-9fde7350b16a","Type":"ContainerDied","Data":"5aa2091c87df3eadcd7b921d3f5ffcf5608df76c63a5cb5752e321d31d4dafcd"}
Nov 04 01:50:14.411744 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.411717    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:14.466063 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.466029    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca\") pod \"046d707a-b0ac-4e60-abbd-9fde7350b16a\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") "
Nov 04 01:50:14.466209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.466127    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca\") pod \"046d707a-b0ac-4e60-abbd-9fde7350b16a\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") "
Nov 04 01:50:14.466209 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.466167    5797 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7\" (UniqueName: \"kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7\") pod \"046d707a-b0ac-4e60-abbd-9fde7350b16a\" (UID: \"046d707a-b0ac-4e60-abbd-9fde7350b16a\") "
Nov 04 01:50:14.466467 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.466442    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca" pod "046d707a-b0ac-4e60-abbd-9fde7350b16a" (UID: "046d707a-b0ac-4e60-abbd-9fde7350b16a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 01:50:14.466533 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.466517    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-openshift-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:50:14.468371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.468340    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca" pod "046d707a-b0ac-4e60-abbd-9fde7350b16a" (UID: "046d707a-b0ac-4e60-abbd-9fde7350b16a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 01:50:14.468371 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.468355    5797 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7" pod "046d707a-b0ac-4e60-abbd-9fde7350b16a" (UID: "046d707a-b0ac-4e60-abbd-9fde7350b16a"). InnerVolumeSpecName "kube-api-access-vg5z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 01:50:14.567517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.567448    5797 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046d707a-b0ac-4e60-abbd-9fde7350b16a-maas-api-xks-ca\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:50:14.567517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:14.567476    5797 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5z7\" (UniqueName: \"kubernetes.io/projected/046d707a-b0ac-4e60-abbd-9fde7350b16a-kube-api-access-vg5z7\") on node \"ip-10-0-140-186.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 01:50:15.287006 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:15.286973    5797 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr"
Nov 04 01:50:15.287207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:15.287015    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zqjvr" event={"ID":"046d707a-b0ac-4e60-abbd-9fde7350b16a","Type":"ContainerDied","Data":"7596ede49282497ecf24318964c783922434510d95ebd1a458fb0c4f954d1340"}
Nov 04 01:50:15.287207 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:15.287069    5797 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7596ede49282497ecf24318964c783922434510d95ebd1a458fb0c4f954d1340"
Nov 04 01:50:22.872518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:22.872473    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ff97f47cd-jh7sr_9f7141ac-7569-418e-9880-8f863b8211d4/authorino/0.log"
Nov 04 01:50:26.251527 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:50:26.251486    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:50:29.714184 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:29.714154    5797 scope.go:122] "RemoveContainer" containerID="8c73484ba3a2ffb41ad6bb7d5b7daeec3b26584d46d11bd6af67e406be3da77f"
Nov 04 01:50:32.304855 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:32.304810    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:50:33.220124 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:33.220083    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:50:34.724292 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:34.724245    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mvf9p_6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc/manager/1.log"
Nov 04 01:50:35.793902 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:35.793872    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jr85b_8931f2f8-db63-47c5-9e05-15b4a0eb9c51/manager/1.log"
Nov 04 01:50:35.996312 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:35.996279    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9s6bw_13f0c474-26fd-4004-8b30-14358246c509/proxy/1.log"
Nov 04 01:50:36.566632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:36.566602    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mvf9p_6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc/manager/0.log"
Nov 04 01:50:36.794154 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:36.794120    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jr85b_8931f2f8-db63-47c5-9e05-15b4a0eb9c51/manager/0.log"
Nov 04 01:50:37.002605 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:37.002573    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9s6bw_13f0c474-26fd-4004-8b30-14358246c509/proxy/0.log"
Nov 04 01:50:37.943900 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:37.943870    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-6xbk9_2e6717d5-8761-408d-934f-ff15dee882b8/revoke-keys/0.log"
Nov 04 01:50:38.473013 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:38.472960    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zqjvr_046d707a-b0ac-4e60-abbd-9fde7350b16a/revoke-keys/0.log"
Nov 04 01:50:38.829772 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:38.829698    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-cs6kr_9a341cc3-ab85-4090-826d-c8d56bb849a3/revoke-keys/0.log"
Nov 04 01:50:39.030632 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:39.030600    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-gsw2f_35506939-7454-4450-9d9b-687b5caaea77/postgres/0.log"
Nov 04 01:50:40.837764 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:40.837733    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ff97f47cd-jh7sr_9f7141ac-7569-418e-9880-8f863b8211d4/authorino/0.log"
Nov 04 01:50:41.060012 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:41.059957    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7l6kd_f493fdd3-b7e8-493f-af2e-df12f9651df0/manager/1.log"
Nov 04 01:50:41.232437 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:41.232392    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-smsgb_b96933e9-892e-41b6-9b0f-9568cb44c538/manager/1.log"
Nov 04 01:50:42.619857 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:42.619802    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7l6kd_f493fdd3-b7e8-493f-af2e-df12f9651df0/manager/0.log"
Nov 04 01:50:42.808959 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:42.808923    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-smsgb_b96933e9-892e-41b6-9b0f-9568cb44c538/manager/0.log"
Nov 04 01:50:43.824517 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:43.824487    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-654z6_517f1a3c-9ae6-4bbb-a412-22101ed0784f/kube-auth-proxy/0.log"
Nov 04 01:50:44.576568 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:44.576537    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-b44x4_fd4a4e92-a390-4fc8-813d-71dff7e4e07c/istio-proxy/0.log"
Nov 04 01:50:44.932113 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:44.932088    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-dw6nw_fe939348-5f96-41ac-b733-2746eb79904d/istio-proxy/0.log"
Nov 04 01:50:45.865627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:45.865594    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-x2k4g_c6b2d0c8-f594-4dc9-b8ff-a3dec860c112/istio-proxy/0.log"
Nov 04 01:50:46.050243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:46.050214    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-597546666c-728zv_0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b/istio-proxy/0.log"
Nov 04 01:50:48.292727 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:48.292690    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qb7m9_270dc588-8548-488f-a219-e745a8d110db/main/0.log"
Nov 04 01:50:48.663361 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:48.663327    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-pqxmd_5d133c42-6f8e-420e-861b-37d8e673161b/main/0.log"
Nov 04 01:50:50.136130 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:50.136098    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ff97f47cd-jh7sr_9f7141ac-7569-418e-9880-8f863b8211d4/authorino/0.log"
Nov 04 01:50:56.250895 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:50:56.250855    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:50:59.620630 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:50:59.620588    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mvf9p_6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc/manager/1.log"
Nov 04 01:51:00.700303 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:00.700275    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jr85b_8931f2f8-db63-47c5-9e05-15b4a0eb9c51/manager/1.log"
Nov 04 01:51:00.909862 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:00.909830    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9s6bw_13f0c474-26fd-4004-8b30-14358246c509/proxy/1.log"
Nov 04 01:51:01.868441 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:01.868407    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-6xbk9_2e6717d5-8761-408d-934f-ff15dee882b8/revoke-keys/0.log"
Nov 04 01:51:02.454496 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:02.454459    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zqjvr_046d707a-b0ac-4e60-abbd-9fde7350b16a/revoke-keys/0.log"
Nov 04 01:51:02.821479 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:02.821386    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-cs6kr_9a341cc3-ab85-4090-826d-c8d56bb849a3/revoke-keys/0.log"
Nov 04 01:51:03.023927 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:03.023899    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-gsw2f_35506939-7454-4450-9d9b-687b5caaea77/postgres/0.log"
Nov 04 01:51:04.131196 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:04.131164    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ff97f47cd-jh7sr_9f7141ac-7569-418e-9880-8f863b8211d4/authorino/0.log"
Nov 04 01:51:04.382699 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:04.382598    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7l6kd_f493fdd3-b7e8-493f-af2e-df12f9651df0/manager/1.log"
Nov 04 01:51:04.578243 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:04.578210    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-smsgb_b96933e9-892e-41b6-9b0f-9568cb44c538/manager/1.log"
Nov 04 01:51:06.298833 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:06.298772    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-654z6_517f1a3c-9ae6-4bbb-a412-22101ed0784f/kube-auth-proxy/0.log"
Nov 04 01:51:07.086262 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:07.086226    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-b44x4_fd4a4e92-a390-4fc8-813d-71dff7e4e07c/istio-proxy/0.log"
Nov 04 01:51:07.455659 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:07.455628    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-dw6nw_fe939348-5f96-41ac-b733-2746eb79904d/istio-proxy/0.log"
Nov 04 01:51:08.454420 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:08.454393    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-x2k4g_c6b2d0c8-f594-4dc9-b8ff-a3dec860c112/istio-proxy/0.log"
Nov 04 01:51:08.641388 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:08.641335    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-597546666c-728zv_0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b/istio-proxy/0.log"
Nov 04 01:51:10.333693 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:10.333645    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qb7m9_270dc588-8548-488f-a219-e745a8d110db/main/0.log"
Nov 04 01:51:10.712220 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:10.712183    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-pqxmd_5d133c42-6f8e-420e-861b-37d8e673161b/main/0.log"
Nov 04 01:51:21.673278 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:21.673245    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:51:26.251854 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:51:26.251811    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:51:29.738586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:29.738557    5797 scope.go:122] "RemoveContainer" containerID="22901fe6c93914712844b459f3e8d8ac95d5715efc3e316a15e4f94cb3d6252f"
Nov 04 01:51:32.354146 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:32.354111    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:51:33.266070 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:51:33.266031    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:51:56.251372 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:51:56.251334    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:52:03.994706 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:03.994662    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:52:04.199930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:04.199886    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:52:24.967578 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:24.967536    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:52:25.178725 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:25.178688    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:52:26.250623 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:52:26.250574    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:52:28.207464 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:28.207427    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x7c8v_bc5258b8-2791-49ac-a271-e914db291c6f/global-pull-secret-syncer/0.log"
Nov 04 01:52:28.368610 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:28.368580    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log"
Nov 04 01:52:28.562045 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:28.561964    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-186.us-east-2.compute.internal_388b6cc67d09cb3e8a1219d1e84089fb/haproxy/0.log"
Nov 04 01:52:32.413835 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:32.413773    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:52:33.311858 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:33.311822    5797 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch6l_d6eb6955-2fed-4ca2-9b74-e278bb5eb79c/konnectivity-agent/0.log" limit=1024
Nov 04 01:52:35.179724 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:35.179693    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ff97f47cd-jh7sr_9f7141ac-7569-418e-9880-8f863b8211d4/authorino/0.log"
Nov 04 01:52:35.215619 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:35.215590    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7l6kd_f493fdd3-b7e8-493f-af2e-df12f9651df0/manager/1.log"
Nov 04 01:52:35.218907 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:35.218884    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7l6kd_f493fdd3-b7e8-493f-af2e-df12f9651df0/manager/0.log"
Nov 04 01:52:35.244652 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:35.244628    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-smsgb_b96933e9-892e-41b6-9b0f-9568cb44c538/manager/1.log"
Nov 04 01:52:35.244809 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:35.244791    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-smsgb_b96933e9-892e-41b6-9b0f-9568cb44c538/manager/0.log"
Nov 04 01:52:37.388188 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.388157    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bw7dk_9693b4e7-8f19-4e24-aac8-4fb293180bae/kube-state-metrics/0.log"
Nov 04 01:52:37.410087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.410061    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bw7dk_9693b4e7-8f19-4e24-aac8-4fb293180bae/kube-rbac-proxy-main/0.log"
Nov 04 01:52:37.435087 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.435067    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bw7dk_9693b4e7-8f19-4e24-aac8-4fb293180bae/kube-rbac-proxy-self/0.log"
Nov 04 01:52:37.468572 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.468549    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bb86d5d94-r4w4m_1a2e54ab-acb8-4611-bf2d-534053133f77/metrics-server/1.log"
Nov 04 01:52:37.469367 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.469350    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bb86d5d94-r4w4m_1a2e54ab-acb8-4611-bf2d-534053133f77/metrics-server/0.log"
Nov 04 01:52:37.532806 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.532783    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29mzn_513cdfcf-ac71-4d24-aca9-1c81edb6f803/node-exporter/0.log"
Nov 04 01:52:37.556090 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.556071    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29mzn_513cdfcf-ac71-4d24-aca9-1c81edb6f803/kube-rbac-proxy/0.log"
Nov 04 01:52:37.580300 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.580275    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29mzn_513cdfcf-ac71-4d24-aca9-1c81edb6f803/init-textfile/0.log"
Nov 04 01:52:37.888185 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.888161    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/prometheus/0.log"
Nov 04 01:52:37.926579 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.926548    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/config-reloader/0.log"
Nov 04 01:52:37.950873 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.950848    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/thanos-sidecar/0.log"
Nov 04 01:52:37.976141 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:37.976118    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/kube-rbac-proxy-web/0.log"
Nov 04 01:52:38.000412 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.000385    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/kube-rbac-proxy/0.log"
Nov 04 01:52:38.029037 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.029010    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/kube-rbac-proxy-thanos/0.log"
Nov 04 01:52:38.055357 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.055333    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c1ed5e-0c1f-4cca-a6e6-fc16bf8ecefd/init-config-reloader/0.log"
Nov 04 01:52:38.088223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.088197    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dxwgn_b5df0493-dbfc-4398-94fa-2d5ec7676197/prometheus-operator/0.log"
Nov 04 01:52:38.128639 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.128608    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dxwgn_b5df0493-dbfc-4398-94fa-2d5ec7676197/kube-rbac-proxy/0.log"
Nov 04 01:52:38.206940 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.206842    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b6c7df88-lkrmc_77b967b9-f2b0-4b23-9484-038568e3a0ea/telemeter-client/0.log"
Nov 04 01:52:38.234117 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.234068    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b6c7df88-lkrmc_77b967b9-f2b0-4b23-9484-038568e3a0ea/reload/0.log"
Nov 04 01:52:38.269627 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.269599    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b6c7df88-lkrmc_77b967b9-f2b0-4b23-9484-038568e3a0ea/kube-rbac-proxy/0.log"
Nov 04 01:52:38.279405 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.279358    5797 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"]
Nov 04 01:52:38.280062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.280007    5797 cpu_manager.go:414] "containerMap: removing container" podUID="046d707a-b0ac-4e60-abbd-9fde7350b16a" containerName="revoke-keys"
Nov 04 01:52:38.280062 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.280061    5797 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046d707a-b0ac-4e60-abbd-9fde7350b16a" containerName="revoke-keys"
Nov 04 01:52:38.280254 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.280189    5797 memory_manager.go:365] "RemoveStaleState removing state" podUID="046d707a-b0ac-4e60-abbd-9fde7350b16a" containerName="revoke-keys"
Nov 04 01:52:38.283475 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.283454    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.295126 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.295104    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"]
Nov 04 01:52:38.382406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.382366    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsdgx\"/\"kube-root-ca.crt\""
Nov 04 01:52:38.382406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.382383    5797 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsdgx\"/\"openshift-service-ca.crt\""
Nov 04 01:52:38.382635 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.382452    5797 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lsdgx\"/\"default-dockercfg-d5q5k\""
Nov 04 01:52:38.409872 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.409845    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-lib-modules\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.410310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.409877    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-sys\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.410310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.409923    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-podres\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.410310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.410066    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-proc\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.410310 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.410099    5797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvb4j\" (UniqueName: \"kubernetes.io/projected/26dfbc49-6b73-4175-acc8-b190096e8ae0-kube-api-access-pvb4j\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.510947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.510863    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-lib-modules\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.510947 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.510899    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-sys\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.510956    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-podres\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.511017    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-proc\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.511041    5797 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvb4j\" (UniqueName: \"kubernetes.io/projected/26dfbc49-6b73-4175-acc8-b190096e8ae0-kube-api-access-pvb4j\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.511045    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-sys\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.511054    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-podres\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511223 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.511017    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-lib-modules\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.511507 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.511238    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26dfbc49-6b73-4175-acc8-b190096e8ae0-proc\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.523586 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.523554    5797 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvb4j\" (UniqueName: \"kubernetes.io/projected/26dfbc49-6b73-4175-acc8-b190096e8ae0-kube-api-access-pvb4j\") pod \"perf-node-gather-daemonset-bkt6n\" (UID: \"26dfbc49-6b73-4175-acc8-b190096e8ae0\") " pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.597406 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.597375    5797 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:38.869052 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.869015    5797 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"]
Nov 04 01:52:38.871933 ip-10-0-140-186 kubenswrapper[5797]: W1104 01:52:38.871907    5797 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26dfbc49_6b73_4175_acc8_b190096e8ae0.slice/crio-4b82d30fec161773f56440bf726809bb8e4aed62592e493c3e93d4c5529b6aff WatchSource:0}: Error finding container 4b82d30fec161773f56440bf726809bb8e4aed62592e493c3e93d4c5529b6aff: Status 404 returned error can't find the container with id 4b82d30fec161773f56440bf726809bb8e4aed62592e493c3e93d4c5529b6aff
Nov 04 01:52:38.989966 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.989934    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n" event={"ID":"26dfbc49-6b73-4175-acc8-b190096e8ae0","Type":"ContainerStarted","Data":"3ce16741f539a0280b5a55c69825c12df68e3183f2a5333a5896d76207fdf5c7"}
Nov 04 01:52:38.990138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.989981    5797 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n" event={"ID":"26dfbc49-6b73-4175-acc8-b190096e8ae0","Type":"ContainerStarted","Data":"4b82d30fec161773f56440bf726809bb8e4aed62592e493c3e93d4c5529b6aff"}
Nov 04 01:52:38.990138 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:38.990043    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:39.009708 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:39.009658    5797 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n" podStartSLOduration=1.009646193 podStartE2EDuration="1.009646193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:39.007870922 +0000 UTC m=+2946.649777173" watchObservedRunningTime="2026-11-04 01:52:39.009646193 +0000 UTC m=+2946.651552444"
Nov 04 01:52:40.825930 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:40.825897    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b8dc999f8-b5bn9_f51cfd3b-b1ee-4361-aef7-d80d21dac86a/console/1.log"
Nov 04 01:52:40.826761 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:40.826741    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b8dc999f8-b5bn9_f51cfd3b-b1ee-4361-aef7-d80d21dac86a/console/0.log"
Nov 04 01:52:42.733506 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:42.733469    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z88mh_b1de55b1-797e-4e61-aeee-624aa4e20ca5/dns/0.log"
Nov 04 01:52:42.757870 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:42.757840    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z88mh_b1de55b1-797e-4e61-aeee-624aa4e20ca5/kube-rbac-proxy/0.log"
Nov 04 01:52:42.784497 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:42.784464    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6tjgh_95384914-9263-45e9-9639-3566767d7559/dns-node-resolver/0.log"
Nov 04 01:52:43.486447 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:43.486419    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dnjzj_22778bfc-2110-48e0-85dd-25e5ca26c161/node-ca/0.log"
Nov 04 01:52:45.005715 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:45.005679    5797 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lsdgx/perf-node-gather-daemonset-bkt6n"
Nov 04 01:52:45.512888 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:45.512832    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-654z6_517f1a3c-9ae6-4bbb-a412-22101ed0784f/kube-auth-proxy/0.log"
Nov 04 01:52:45.645502 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:45.645469    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-b44x4_fd4a4e92-a390-4fc8-813d-71dff7e4e07c/istio-proxy/0.log"
Nov 04 01:52:45.708349 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:45.708316    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-dw6nw_fe939348-5f96-41ac-b733-2746eb79904d/istio-proxy/0.log"
Nov 04 01:52:45.872774 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:45.872743    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-x2k4g_c6b2d0c8-f594-4dc9-b8ff-a3dec860c112/istio-proxy/0.log"
Nov 04 01:52:45.902048 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:45.902025    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-597546666c-728zv_0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b/istio-proxy/0.log"
Nov 04 01:52:48.168871 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:48.168843    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z5jfh_f132805a-066a-4d03-bcb3-90c1dc9dc73e/serve-healthcheck-canary/0.log"
Nov 04 01:52:49.033955 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:49.033926    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lbqr9_bf31dccb-516a-4a97-aa86-9277ec17154d/kube-rbac-proxy/0.log"
Nov 04 01:52:49.059518 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:49.059487    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lbqr9_bf31dccb-516a-4a97-aa86-9277ec17154d/exporter/0.log"
Nov 04 01:52:49.113695 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:49.113663    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lbqr9_bf31dccb-516a-4a97-aa86-9277ec17154d/extractor/0.log"
Nov 04 01:52:50.286729 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:50.286692    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-654z6_517f1a3c-9ae6-4bbb-a412-22101ed0784f/kube-auth-proxy/0.log"
Nov 04 01:52:50.365416 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:50.365391    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-b44x4_fd4a4e92-a390-4fc8-813d-71dff7e4e07c/istio-proxy/0.log"
Nov 04 01:52:50.401193 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:50.401169    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-dw6nw_fe939348-5f96-41ac-b733-2746eb79904d/istio-proxy/0.log"
Nov 04 01:52:50.499491 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:50.499463    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-x2k4g_c6b2d0c8-f594-4dc9-b8ff-a3dec860c112/istio-proxy/0.log"
Nov 04 01:52:50.516286 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:50.516260    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-597546666c-728zv_0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b/istio-proxy/0.log"
Nov 04 01:52:52.195923 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:52.195897    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8ff7fd4ff-hxltx_3dba5003-a608-45f0-bee9-a0597e2d39b4/manager/1.log"
Nov 04 01:52:52.196526 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:52.196506    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8ff7fd4ff-hxltx_3dba5003-a608-45f0-bee9-a0597e2d39b4/manager/0.log"
Nov 04 01:52:52.220908 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:52.220884    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8ff7fd4ff-hxltx_3dba5003-a608-45f0-bee9-a0597e2d39b4/copy-manifests/0.log"
Nov 04 01:52:53.353869 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:53.353837    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-775c47d899-9l8hb_bd717794-dd18-4bb7-8dfe-222545f12ff7/manager/1.log"
Nov 04 01:52:53.354786 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:53.354770    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-775c47d899-9l8hb_bd717794-dd18-4bb7-8dfe-222545f12ff7/manager/0.log"
Nov 04 01:52:53.379140 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:53.379114    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-7nshw_e34d3f92-7bb9-4f14-8a2e-bd362ae0cfb7/jobset-operator/0.log"
Nov 04 01:52:54.106961 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:54.106934    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mvf9p_6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc/manager/1.log"
Nov 04 01:52:54.107428 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:54.107412    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mvf9p_6d6e56b0-ddd0-4e20-b6c3-4b56e51350fc/manager/0.log"
Nov 04 01:52:54.943593 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:54.943556    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c488b8c84-654z6_517f1a3c-9ae6-4bbb-a412-22101ed0784f/kube-auth-proxy/0.log"
Nov 04 01:52:55.031225 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:55.031189    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-b44x4_fd4a4e92-a390-4fc8-813d-71dff7e4e07c/istio-proxy/0.log"
Nov 04 01:52:55.072027 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:55.071975    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-dw6nw_fe939348-5f96-41ac-b733-2746eb79904d/istio-proxy/0.log"
Nov 04 01:52:55.182861 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:55.182824    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-848f6bf8bc-x2k4g_c6b2d0c8-f594-4dc9-b8ff-a3dec860c112/istio-proxy/0.log"
Nov 04 01:52:55.201867 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:55.201763    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-597546666c-728zv_0ac6cf92-f241-4b0c-98e6-5ca2a56b9f9b/istio-proxy/0.log"
Nov 04 01:52:55.809289 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:55.809251    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jr85b_8931f2f8-db63-47c5-9e05-15b4a0eb9c51/manager/1.log"
Nov 04 01:52:56.261247 ip-10-0-140-186 kubenswrapper[5797]: E1104 01:52:56.261203    5797 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbqr9" containerName="extractor"
Nov 04 01:52:56.417439 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:56.417404    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jr85b_8931f2f8-db63-47c5-9e05-15b4a0eb9c51/manager/0.log"
Nov 04 01:52:56.443794 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:56.443759    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9s6bw_13f0c474-26fd-4004-8b30-14358246c509/proxy/0.log"
Nov 04 01:52:56.445032 ip-10-0-140-186 kubenswrapper[5797]: I1104 01:52:56.445012    5797 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9s6bw_13f0c474-26fd-4004-8b30-14358246c509/proxy/1.log"